Barcode readers with 3d camera(s)

ABSTRACT

Methods and systems include using three-dimensional imaging apparatuses to capture three-dimensional images and analyze resulting three-dimensional image data to enhance captured two-dimensional images for scanning related processes such as object identification, symbology detection, object recognition model training, and identifying improper scan attempts or other actions performed by an operator. Imaging systems such as bi-optic readers, handheld scanners, and machine vision systems are described using three-dimensional imaging apparatuses and described capturing three-dimensional images and using with captured two-dimensional images.

BACKGROUND

Symbology readers, such as handheld barcode readers and bi-opticreaders, use two-dimensional images to capture images of objects, for avariety of purposes. In many warehousing, distribution, and retailenvironments, these symbology readers capture 2D image data which isused to conduct barcode decoding operations. However, in some instancesit is desirable to use the 2D image data for other purposes. Forexample, a bi-optic reader may capture 2D image data from its 2D imagerand use that image data to help determine an off-platter weighingcondition. In other instances, the bi-optic reader may use 2D image datato help determine the nature of the product being scanned. In yet otherinstances, the bi-optic reader may use 2D image data to help detect anoccurrence of a sweethearting or a spoofing event.

While the use of 2D imagers and 2D image analysis has existed inconnection with the aforementioned, and other, purposes, detectionsuccess rate could still be improved. Moreover, image analysis that cantake place is limited by the fact that the captured image data istwo-dimensional. Accordingly, there is a continued need to evolve anddevelop bi-optic barcode readers, and other symbology readers, tofurther increase reader performance and to provide readers with new, asyet unavailable functionality.

SUMMARY

In an embodiment, the present invention is a method of barcode scanningusing a barcode reader. The method includes capturing, using atwo-dimensional (2D) imaging apparatus within the barcode reader andhaving a first field of view (FOV), a 2D image of a first environmentappearing within the first FOV and storing 2D image data correspondingto the 2D image. The method further includes capturing, using athree-dimensional (3D) imaging apparatus associated with the barcodereader and having a second FOV that overlaps at least partially with thefirst FOV, a 3D image of a second environment appearing within thesecond FOV and storing 3D image data corresponding to the 3D image. Themethod further includes identifying one or more 3D image features withinthe second environment from the 3D image data; and enhancing the 2Dimage data, resulting in enhanced 2D image data, by correlating the oneor more 3D image features to at least one or more 2D image features inthe 2D image data. From there, the method includes processing theenhanced 2D image data to at least one of (a) decode a barcode capturedwithin the enhanced 2D image data, (b) train an object recognition modelwith the enhanced 2D image data, (c) recognize an object within theenhanced 2D image data, (d) identify an action performed by an operatorof the barcode reader, and (e) change at least one parameter associatedwith the barcode reader operation.

In a variation of this embodiment, the 3D image data comprises 3D pointcloud data and the one or more 3D image features includes one or moregeometric features of an object presented within the second FOV. Inanother variation of this method, the 3D image data comprises 3D pointcloud data and the one or more 3D image features includes a color or acolor gradation corresponding an object presented within the second FOV.In yet another variation of the embodiment, identifying the one or more3D image features includes identifying the one or more 3D image featuresthat are positioned within a predetermined distance range away from the3D imaging apparatus. In some variations, enhancing the 2D image datafurther includes filtering the at least one or more 2D image featuressuch that the processing the enhanced 2D image data excludes processingof image data associated with the at least one or more 2D imagefeatures. In some variations, enhancing the 2D image data furtherincludes filtering the at least one or more 2D image features such thatthe processing the enhanced 2D image data is limited to processing ofimage data associated with the at least one or more 2D image features.In some variations, the barcode reader is a stationary barcode readerconfigured to be positioned within a workstation and operated by theoperator, and the predetermined distance range away from the 3D imagingapparatus extends from the 3D imaging apparatus to an edge of theworkstation proximate the operator. In some variations, the barcodereader is a bi-optic barcode reader having a product-scanning region,and the predetermined distance range away from the 3D imaging apparatusextends from the 3D imaging apparatus to a distal boundary of theproduct-scanning region.

In another variation of the embodiment, the one or more 3D imagefeatures includes at least one of (i) at least a portion of anoperator's hand, and (ii) an object grasped by the operator's hand.

In another variation of the embodiment, the processing the enhanced 2Dimage data includes the identifying the action performed by theoperator, and responsive to the action performed by the operator beingidentified as one of presenting an object within a product-scanningregion and presenting the object proximate to the product-scanningregion, and further responsive to no barcode being detected within atleast one of the 2D image data and the enhanced 2D image data,generating an alert suitable for signaling a potential theft event.

In another variation of this embodiment, processing the enhanced 2Dimage data includes identifying the action performed by the operator,and responsive to the action performed by the operator being identifiedas one of presenting an object within a product-scanning region andpresenting the object proximate to the product-scanning region, anddetecting a partially covered barcode or fully covered barcode on theobject within at least one of the 2D image data and the enhanced 2Dimage data, generating an alert suitable for signaling a potential theftevent.

In another variation of this embodiment, the at least one parameterassociated with the barcode reader operation is an exposure time of thebarcode reader, an illumination pulse duration of the barcode reader, afocus position of the barcode reader, an imaging zoom level of thebarcode reader, and an illumination source of the barcode reader. In anexample, the illumination source is a diffuse illumination source or adirect illumination source.

In another variation of this embodiment, in response to identifying theone or more 3D image features within the second environment from the 3Dimage data, adjusting an illumination brightness of the barcode readerprior to capturing the 2D image of the first environment.

In another embodiment, the present invention is a method processing datausing a barcode reader. The method includes capturing, using atwo-dimensional (2D) imaging apparatus within the barcode reader andhaving a first field of view (FOV), a 2D image of a first environmentappearing within the first FOV and storing 2D image data correspondingto the 2D image. The method further includes capturing, using athree-dimensional (3D) imaging apparatus associated with the barcodereader and having a second FOV that overlaps at least partially with thefirst FOV, a 3D image of a second environment appearing within thesecond FOV and storing 3D image data corresponding to the 3D image. Themethod further includes identifying one or more 2D image features withinthe first environment from the 2D image data; and enhancing the 3D imagedata, resulting in enhanced 3D image data, by correlating the one ormore 2D image features to at least one or more 3D image features in the3D image data. From there, the method includes processing the enhanced3D image data to at least one of (a) train an object recognition modelwith the enhanced 3D image data, (b) recognize an object within theenhanced 3D image data, (c) identify an action performed by a user ofthe barcode reader, and (d) change at least one parameter associatedwith the 3D imaging apparatus.

In a variation of this embodiment, the 2D image data comprises one ofmonochrome image data, grayscale image data, and polychrome image data,and the one or more 2D image features includes at least one of a barcodeand one or more geometric features of an object presented within thefirst FOV.

In another variation of this embodiment, the one or more 2D imagefeatures includes the barcode, and the enhancing the 3D image dataincludes mapping a location of the barcode from the 2D image data to the3D image data. In another variation of this embodiment, the 2D imagedata comprises polychrome image data, the one or more 2D image featuresincludes the one or more geometric features of the object presentedwithin the first FOV, and enhancing the 3D image data includes mappingat least a portion of the polychrome image data to the 3D image databased at least in part on the one or more geometric features of theobject presented within the first FOV. In another variation of thisembodiment, enhancing the 3D image data further includes filtering theat least one or more 3D image features such that the processing theenhanced 3D image data excludes processing of image data associated withthe at least one or more 3D image features. In another variation of thisembodiment, enhancing the 3D image data further includes filtering theat least one or more 3D image features such that the processing theenhanced 3D image data is limited to processing of image data associatedwith the at least one or more 3D image features. In another variation ofthis embodiment, enhancing the 3D image data further includes filteringthe 3D image databased on a predetermined distance range away from the3D imaging apparatus.

In another variation of this embodiment, the barcode reader is astationary barcode reader configured to be positioned within aworkstation and operated by the operator, and the predetermined distancerange away from the 3D imaging apparatus extends from the 3D imagingapparatus to an edge of the workstation proximate the operator. Inanother variation of this embodiment, the barcode reader is a bi-opticbarcode reader having a product-scanning region, and the predetermineddistance range away from the 3D imaging apparatus extends from the 3Dimaging apparatus to a distal boundary of the product-scanning region.

In another variation of this embodiment, the one or more 2D imagefeatures includes at least one of (i) at least a portion of anoperator's hand, and (ii) an object grasped by the operator's hand.

In another variation of this embodiment, processing the enhanced 3Dimage data includes the identifying the action performed by theoperator, and responsive to the action performed by the operator beingidentified as one of presenting an object within a product-scanningregion and presenting the object proximate to the product-scanningregion, and further responsive to no barcode being detected within the2D image data generating an alert suitable for signaling a potentialtheft event.

In another variation of this embodiment, identifying the one or more 2Dimage features includes identifying environmental features on the 2Dimage, wherein the environmental features are features in the imageoutside of an object presented within the first FOV; converting theenvironmental features into masking features configured to cover theidentified environmental features in the 2D image; and identifying themasking features as the one or more 2D image features.

In another variation of this embodiment, identifying one or more 2Dimage features comprises: identifying a barcode for the object in the 2Dimage data; decoding the barcode to generate barcode payload data anddetermining object identification from the barcode payload data; anddetermining, from the objecting identification, the one or more 2D imagefeatures.

In another variation of this embodiment, processing the enhanced 3Dimage data to train the object recognition model with the enhanced 3Dimage data includes: identifying a barcode in the 2D image data,determining a barcode detection event timeframe, and training the objectrecognition model with the enhanced 3D image data corresponding to thebarcode detection event timeframe; or identifying a barcode in the 2Dimage data, identifying the object within the enhanced 3D image datacorresponding to the barcode in the 2D image data, and upon identifyingother object in the 3D image not corresponding to the barcode, removingthe other objects from the enhanced 3D image data before training theobject recognition model with the enhanced 3D image data.

In another variation of this embodiment, the at least one parameterassociated with the 3D imaging apparatus comprises a projected amount ofillumination of the 3D imaging apparatus, a projected direction ofillumination of the 3D imaging apparatus, or a source of illuminationfor the 3D imaging apparatus.

In another embodiment, the present invention is a method of identifyinga proper scanning of an object or an improper scanning of the objectusing a barcode reader. The method includes capturing, using atwo-dimensional (2D) imaging apparatus within the barcode reader andhaving a first field of view (FOV), a 2D image of a first environmentappearing within the first FOV and storing 2D image data correspondingto the 2D image. The method further includes capturing, using athree-dimensional (3D) imaging apparatus associated with the barcodereader and having a second FOV that overlaps at least partially with thefirst FOV, a 3D image of a second environment appearing within thesecond FOV and storing 3D image data corresponding to the 3D image. Themethod further includes determining a first object identification of theobject using the 2D image data and determining a second objectidentification of the object using the 3D image data. From there, themethod includes comparing the first object identification with thesecond object identification, and determining (a) the proper scanning ofthe object when the first object identification matches the secondobject identification and (b) the improper scanning of the object whenthe first object identification does not match the second objectidentification.

In a variation of this embodiment, determining the first objectidentification of the object using the 2D image data comprises:identifying a barcode for the object in the 2D image data; and decodingthe barcode to generate barcode payload data and determining the firstobject identification from the barcode payload data.

In a variation of this embodiment, determining the first objectidentification of the object using the 2D image data comprises:providing the 2D image data to a trained object recognition model; andproducing, using the trained object recognition model, the first objectidentification of the object.

In a variation of this embodiment, determining the second objectidentification of the object using the 3D image data comprises:providing the 3D image data to a trained object recognition model; andproducing, using the trained object recognition model, the second objectidentification of the object.

In a variation of this embodiment, before determining the first objectidentification of the object using the 2D image data, the method furtherincludes: comparing the 3D image data to the 2D image data; and removingenvironmental features outside of the object from the 2D image databased on the 3D image data.

In a variation of this embodiment, before determining the second objectidentification of the object using the 3D image data, the methodincludes: comparing the 3D image data to the 2D image data; and removingenvironmental features outside of the object from the 3D image databased on the 2D image data.

In a variation of this embodiment, determining the second objectidentification of the object using the 3D image data comprises:determining one or more color features of the object from the 3D imagedata; and determining the second object identification from the one ormore color features.

In a variation of this embodiment, the one or more color featurescomprise a color of the object.

In a variation of this embodiment, the one or more color featurescomprise a color gradient of the object.

In a variation of this embodiment, wherein determining the second objectidentification of the object using the 3D image data comprises:determining one or more geometric features of the object from the 3Dimage data; and determining the second object identification from theone or more geometric features.

In a variation of this embodiment, determining the first objectidentification of the object using the 2D image data comprises:identifying a barcode for the object in the 2D image data; and decodingthe barcode to generate barcode payload data and determining the firstobject identification from the barcode payload data.

In a variation of this embodiment, the 3D image data includes a pointcloud comprising a plurality of data points, each of the data pointshaving a distance value associated with a distance from the 3D imagingapparatus, and determining the one or more geometric features of theobject from the 3D image data is based on a first subset of the 3D imagedata and is not based on a second subset of the 3D image data, the firstsubset of the 3D image data being associated with a first subset of thedata points having the respective distance value associated with thedistance from the 3D imaging apparatus being within a predeterminedrange, the second subset of the 3D image data being associated with asecond subset of the data points having the respective distance valueassociated with the distance from the 3D imaging apparatus being outsideof the predetermined range.

In a variation of this embodiment, responsive to the determining (a) theproper scanning of the object, the method further comprises processing atransaction log to include the data associated with the object, andresponsive to the determining (b) the improper scanning of the object,the method further comprises at least one of (i) generating an alertsuitable for signaling a potential theft event, and (ii) processing thetransaction log to not include the data associated with the object.

In another embodiment, a method of identifying a an improper scanning ofthe object using a barcode reader, the method includes: capturing, usinga two-dimensional (2D) imaging apparatus within the barcode reader andhaving a first field of view (FOV), a 2D image of a first environmentappearing within the first FOV and storing 2D image data correspondingto the 2D image; capturing, using a three-dimensional (3D) imagingapparatus associated with the barcode reader and having a second FOVthat overlaps at least partially with the first FOV, a 3D image of asecond environment appearing within the second FOV and storing 3D imagedata corresponding to the 3D image; identifying a scannable object usingthe 3D image data; and upon failing to determine an objectidentification of the object using the 2D image data, determining animproper scanning of the object and generating an alarm signal.

In another embodiment, the present invention is a method of operating abarcode reader. The method includes capturing, using a three-dimensional(3D) imaging apparatus within the barcode reader and having a first FOV,a 3D image of a first environment appearing within the first FOV andstoring 3D image data corresponding to the 3D image; performing facialrecognition on the 3D image data and identifying a presence of facialdata in the 3D image data; and in response to the identifying thepresence of the facial data, adjust at least one operating parameter ofa two-dimensional (2D) imaging apparatus within the barcode reader.

In a variation of this embodiment, the barcode reader is a presentationbarcode reader, and the adjust the at least one operating parameter ofthe 2D imaging apparatus includes reducing intensity of at least one ofan illumination assembly and an aiming assembly.

In a variation of this embodiment, wherein the barcode reader is apresentation barcode reader, and wherein the adjust the at least oneoperating parameter of the 2D imaging apparatus includes preventingactivation of at least some portion of the 2D imaging apparatus until asubsequent performing of facial recognition on a subsequent 3D imagedata associated with a subsequent 3D image fails to identify anotherpresence of facial data in the subsequent 3D image data.

In a variation of this embodiment, the method further includes capturinga 2D image of an object, using the 2D imaging apparatus adjustedaccording to the at least one operating parameter, and decoding abarcode in the 2D image to identify the object.

In a variation of this embodiment, identifying presence of the facialdata in the 3D image data comprises determining a position of the facialdata in a first FOV of the 3D imaging apparatus, and wherein theadjusting the operating parameters of the 2D imaging apparatus comprisesadjusting the operating parameters based on the position of the facialdata.

In a variation of this embodiment, the operating parameters comprises asecond FOV of the 2D imaging apparatus.

In a variation of this embodiment, the operating parameters comprises afocal distance of the 2D imaging apparatus.

In a variation of this embodiment, the at least one operating parameterof the 2D imaging apparatus is an exposure time, an illumination pulseduration, or an imaging zoom level.

In another embodiment, a method of operating a barcode reader, themethod includes: capturing, using a two-dimensional (2D) imagingapparatus within the barcode reader and having a first field of view(FOV), a 2D image of a first environment appearing within the first FOVand storing 2D image data corresponding to the 2D image; performingfacial recognition on the 2D image data and identifying a presence offacial data in the 2D image data; capturing, using a three-dimensional(3D) imaging apparatus within the barcode reader and having a first FOV,a 3D image of a second environment appearing within the first FOV andstoring 3D image data corresponding to the 3D image; and in response toidentifying one or more 3D image features associated with the facialdata in 2D image data, performing at least one of (a) determining adistance of the facial data from the barcode reader and selectivelydisabling/enabling scanning of the barcode reader based on the distance,(b) determining anthropometric data for the facial data and determiningif facial data is from a person or not, and (c) adjust at least oneoperating parameter of the 2D imaging apparatus within the barcodereader.

In a variation of this embodiment, the at least one operating parameterof the 2D imaging apparatus is an exposure time, an illumination pulseduration, a focus position, or an imaging zoom level.

In a variation of this embodiment, identifying presence of the facialdata in the 3D image data comprises determining a position of the facialdata in a first FOV of the 3D imaging apparatus, and the adjusting theoperating parameters of the 2D imaging apparatus comprises adjusting theoperating parameters based on the position of the facial data.

In another embodiment, the present invention is a method of operatingpoint-of-sale scanning station having a barcode reader. The methodincludes: capturing, using a three-dimensional (3D) imaging apparatusassociated with the point-of-sale scanning station and having a firstFOV, a 3D image of a first environment appearing within the first FOVand storing 3D image data corresponding to the 3D image. The methodfurther includes performing facial recognition on the 3D image data andidentifying a presence of facial data in the 3D image data; andperforming a facial identification on the facial data and authenticatingthe facial identification; and responsive to authenticating the facialidentification, at least one of (a) capturing a two-dimensional (2D)image of an object using a 2D imaging apparatus within the barcodereader and decoding a barcode in the 2D image to identify the object and(b) satisfying a release condition to either prevent a decode of abarcode captured in image of 2D image or to prevent adding asubsequently scanned item to a transaction log of scanned items.

In a variation on this embodiment, authenticating the facialidentification comprises comparing the facial identification to anauthorized user database. In a variation on this embodiment, wherein theauthenticating the facial identification comprises determining that thefacial data is at an acceptable position within the first FOV of the 3Dimaging apparatus. In a variation on this embodiment, before theperforming facial recognition on the 3D image data and the identifyingthe presence of the facial data in the 3D image data, the methodincludes: identifying environmental features in the 3D image data, theenvironmental features being features in the 3D image outside of theobject; and removing environmental features from the 3D image data.

In another embodiment, a machine vision method includes capturing, usinga two-dimensional (2D) imaging apparatus, a 2D image of an object,identifying a barcode of the object in the 2D image, and determining oneor more three-dimensional (3D) object features of the object from thebarcode in the 2D image. The method further includes capturing, using athree-dimensional (3D) imaging apparatus of the machine vision system, a3D image of an environment and storing 3D image data corresponding tothe 3D image. The method includes examining the 3D image data for apresence of the one or more 3D object features; in response todetermining that at least one of the one or more 3D object features isabsent from the 3D image data, providing a digital fault detectionsignal to a user of the machine vision system and in response todetermining that at least one of the one or more 3D object features ispresent from the 3D image data changing at least one parameterassociated with the machine vision system.

In some variations of this method, determining the one or more 3D objectfeatures of the object from the barcode in the 2D image, comprises:decoding the barcode to generate barcode payload data and determiningobject identification from the barcode payload data; and determining,from the objecting identification, the one or more 3D object features ofthe object. In some variations of this method, determining the one ormore 3D object features of the object from the barcode in the 2D image,comprises: determining an orientation of the object from a location ofthe barcode in the 2D image; and determining, from the orientation ofthe object, the one or more 3D object features as a subset of available3D object features within the first FOV.

In some variations of this method, the one or more 3D object features isat least one of a dimensional feature and a shape feature.

In another variation of this embodiment, changing at least one parameterassociated with the machine vision system comprises changing an exposuretime of a 2D imaging apparatus of the machine vision system, anillumination pulse duration of an illumination assembly of the machinevision system, a focal position of the 2D imaging apparatus of themachine vision system, an imaging zoom level of the 2D imagingapparatus, an illumination brightness, an illumination wavelength, or anillumination source of the machine vision system. In some suchembodiments, the illumination source is a diffuse illumination source ora direct illumination source. In some such embodiments, changing theillumination source comprises changing from an illumination sourceemitting at a first wavelength to an illumination source emitting at asecond wavelength, different than the first wavelength.

In another embodiment, the present invention is a scanning station. Thescanning station includes a two-dimensional (2D) imaging apparatusconfigured to: capture a 2D image of an object in a field of view (FOV)of the 2D imaging apparatus; identify a barcode in the 2D image; andidentify the object from a barcode payload. The scanning station furtherincludes a 3D imaging apparatus configured to: capture a 3D image of theobject in a FOV of the 3D imaging apparatus; and generate 3D image datafrom the 3D image. The scanning station further includes a processor andmemory storing instructions that, when executed, cause the processor to:identify one or more 3D object features from the 3D image data; evaluatethe one or more 3D object features against an identity of the object;and responsive to evaluating the one or more 3D object features againstthe identity of the object, adjust an operating parameter of thescanning station.

In a variation of this embodiment, the 3D image data comprises 3D pointcloud data and the one or more 3D object features comprise at least oneof geometric features of the object, color of the object, colorgradation of the object. In another variation of this embodiment, the 3Dimage data comprises 3D point cloud data and the one or more 3D objectfeatures comprise a position of the object in the FOV of the 3D imagingapparatus. In some variations, the memory further stores instructionsthat, when executed, cause the processor to: determine if the positionof the object in the FOV or the 3D imaging apparatus is in an acceptablerange for capturing the 2D image of the object using the 2D imagingapparatus; and in response to the position of the object in the FOV ofthe 3D imaging apparatus not being in the acceptable range, abstain fromcapturing a subsequent 2D image by the 2D imaging apparatus, until arelease condition is satisfied.

In another variation of this embodiment, the operating parametercomprises at least one of an illumination intensity of an illuminationapparatus in the 2D imaging apparatus, a FOV of the 2D imagingapparatus, and a focal distance of the 2D imaging apparatus. In anothervariation of this embodiment, the scanning station is a bi-optic scannerhaving a tower portion and a platter portion. In some variations, the 3Dimaging apparatus is one of the tower portion and the platter portionand the 2D imaging apparatus is in the other of the tower portion andthe platter portion. In some variations, the 3D imaging apparatus andthe 2D image apparatus are both in one of the tower portion and theplatter portion.

In some variations, the scanning station further includes a weighplatter within the platter portion, the weigh platter configured tomeasure, via a weighing module, a weight of an object placed on theweigh platter, the weigh platter having a weighing surface, and thememory further stored instructions that, when executed, cause theprocessor to: determine, from the 3D image data, a position of theobject relative to the weigh platter; and in response to the position ofthe object relative to the weigh platter being in a weigh platter faultposition, modifying operation of the weighing module from a first stateto a second state, until a release condition is satisfied. In somevariations, the first state of the weighing module permits reporting ofthe weight of the object placed on the weigh platter, and wherein thesecond state of the weighing module prevents reporting of the weight ofthe object placed on the weigh platter. In some variations, the weighplatter fault position comprises an overhang position, where at least aportion of the object overhangs the weigh platter. In some variations,the weigh platter fault position comprises a suspended position, wherethe object is suspended, at least partially, above the weigh platter.

In some variations of this embodiment, the scanning station furtherincludes a weigh platter within the platter portion, the weigh platterconfigured to measure, via a weighing module, a weight of an objectplaced on the weigh platter, the weigh platter having a weighingsurface, wherein the memory further stored instructions that, whenexecuted, cause the processor to: detect, from the 3D image data, anoperator's hand being in contact with the object; and in response to adetection of the operator's hand being in contact with the object,modifying operation of the weighing module from a first state to asecond state, until a release condition is satisfied.

In another embodiment, the present invention is a scanning station. Thescanning station includes a 2D imaging apparatus configured to capture a2D image of an object in a field of view of the 2D imaging apparatus andgenerate 2D image data from the 2D image; and a 3D imaging apparatusconfigured to capture a 3D image of the object in a field of view of the3D imaging apparatus and generate 3D image data from the 3D image. Thescanning station further includes a processor and memory storinginstructions that, when executed, cause the processor to: compare the 3Dimage data to the 2D image data and performing an authentication processon the object.

In a variation of this embodiment, the memory further storesinstructions that, when executed, cause the processor to: determine afirst object identification of the object using the 2D image data;determine a second object identification of the object using the 3Dimage data; and compare the first object identification to the secondobject identification and determine (a) a proper scanning of the objectwhen the first object identification matches the second objectidentification and (b) an improper scanning of the object when the firstobject identification does not match the second object identification.In another variation of this embodiment, the memory further storesinstructions that, when executed, cause the processor to determine thefirst object identification of the object using the 2D image data by:identifying a barcode for the object in the 2D image data; and decodingthe barcode to generate barcode payload data and determining the firstobject identification from the barcode payload data. In anothervariation of this embodiment, the memory further stores instructionsthat, when executed, cause the processor to determine the first objectidentification of the object using the 2D image data by: providing the2D image data to a trained object recognition model; and producing,using the trained object recognition model, the first objectidentification of the object.

In some variations, the memory further stores instructions that, whenexecuted, cause the processor to determine the second objectidentification of the object using the 3D image data by: providing the3D image data to a trained object recognition model; and producing,using the trained object recognition model, the second objectidentification of the object. In some variations, the memory furtherstores instructions that, when executed, cause the processor to, beforedetermining the first object identification of the object using the 2Dimage data: compare the 3D image data to the 2D image data; and removeenvironmental features outside of the object from the 2D image databased on the 3D image data. In some variations, the memory furtherstores instructions that, when executed, cause the processor to, beforedetermining the second object identification of the object using the 3Dimage data: compare the 3D image data to the 2D image data; and removeenvironmental features outside of the object from the 3D image databased on the 2D image data. In some variations, the memory furtherstores instructions that, when executed, cause the processor todetermine the second object identification of the object using the 3Dimage data by: determining one or more color features of the object fromthe 3D image data; and determining the second object identification fromthe one or more color features. In some variations, the one or morecolor features comprise a color of the object. In some variations, theone or more color features comprise a color gradient of the object. Insome variations, the memory further stores instructions that, whenexecuted, cause the processor to determine the second objectidentification of the object using the 3D image data by: determining oneor more geometric features of the object from the 3D image data; anddetermining the second object identification from the one or moregeometric features.

In some variations, determining the first object identification of theobject using the 2D image data comprises: identifying a barcode for theobject in the 2D image data; and decoding the barcode to generatebarcode payload data and determining the first object identificationfrom the barcode payload data.

In some variations, the 3D image data includes a point cloud comprisinga plurality of data points, each of the data points having a distancevalue associated with a distance from the 3D imaging apparatus, anddetermining the one or more geometric features of the object from the 3Dimage data is based on a first subset of the 3D image data and is notbased on a second subset of the 3D image data, the first subset of the3D image data being associated with a first subset of the data pointshaving the respective distance value associated with the distance fromthe 3D imaging apparatus being within a predetermined range, the secondsubset of the 3D image data being associated with a second subset of thedata points having the respective distance value associated with thedistance from the 3D imaging apparatus being outside of thepredetermined range.

In some variations, the memory further stores instructions that, whenexecuted, cause the processor to, responsive to the determining (a) theproper scanning of the object, causing the processor to processing atransaction log to include the data associated with the object, andresponsive to the determining (b) the improper scanning of the object,at least one of (i) generate an alert suitable for signaling a potentialtheft event, and (ii) process the transaction log to not include thedata associated with the object.

In some variations, the memory further stores instructions that, whenexecuted, cause the processor to: determine, from the 3D image data, ascanning direction of the object relative to the 2D imaging apparatus;and in response to the scanning direction being an improper scanningdirection, preventing capturing of the 2D image by the 2D imagingapparatus, until a release condition is satisfied.

In some variations, the object is produce, and the memory further storesinstructions that, when executed, cause the processor to: determine, asa first object identification of the object using the 2D image data, oneor more colors of the object; determine, as a second objectidentification of the object using the 3D image data, a shape ordimension of the object; and compare the first object identification tothe second object identification and determine a type of the produce.

In some variations, the object is produce, and the memory further storesinstructions that, when executed, cause the processor to: determine, asa first object identification of the object using the 2D image data, oneor more colors of the object; determine, as a second objectidentification of the object using the 3D image data, a shape ordimension of the object; and compare the first object identification tothe second object identification and determine a listing potential typesof the produce; and present the listing to a user of the scanningstation for selection.

In some variations, the memory further stores instructions that, whenexecuted, cause the processor to: determine, as a first objectidentification of the object using the 2D image data, a presence of apartially-decodable barcode and determine a listing of potential objectmatches from the partially-decodable barcode; determine, as a secondobject identification of the object using the 3D image data, a shape ordimension of the object; and compare the first object identification tothe second object identification and determine if one or more of thelisting of potential object matches corresponds to the shape ordimension indicated in the second object identification from the 3Dimage data.

In another embodiment, the present invention is a scanning station. Thescanning station includes a two-dimensional (2D) imaging apparatusconfigured to capture a 2D image of an object in a field of view (FOV)of the 2D imaging apparatus and to identify a barcode in the 2D imagefor identifying the object from a barcode payload. The scanning stationfurther includes a 3D imaging apparatus configured to capture a 3D imageof the object in a FOV of the 3D imaging apparatus and generate 3D imagedata from the 3D image. The scanning station further includes aprocessor and memory storing instructions that, when executed, cause theprocessor to: capture the 3D image of the object and generate the 3Dimage data; identify one or more 3D object features of the object fromthe 3D image data; and perform one of (a) training an object recognitionmodel with the 3D image data or (b) object recognition using the 3Dimage data.

In another embodiment, the prevent invention is a system. The systemincludes a two-dimensional (2D) imaging apparatus having a first fieldof view (FOV) and configured to capture a 2D image of a firstenvironment appearing within the first FOV, the 2D image being stored as2D image data corresponding to the 2D image. The system further includesa three-dimensional (3D) imaging apparatus having a second FOV thatoverlaps at least partially with the first FOV, the 3D imaging apparatusbeing configured to capture a 3D image of a second environment appearingwithin the second FOV, the 3D image being stored as 3D image datacorresponding to the 3D image. The system further includes a processorand memory storing instructions that, when executed, cause the processorto: identify one or more 3D image features within the second environmentfrom the 3D image data; enhance the 2D image data, resulting in enhanced2D image data, by correlating the one or more 3D image features to atleast one or more 2D image features in the 2D image data; and processthe enhanced 2D image data to at least one of (a) decode a barcodecaptured within the enhanced 2D image data, (b) train an objectrecognition model with the enhanced 2D image data, (c) recognize anobject within the enhanced 2D image data, and (d) identify an actionperformed by an operator of the barcode reader.

In a variation of this embodiment, the 3D image data comprises 3D pointcloud data and the one or more 3D image features includes one or moregeometric features of an object presented within the second FOV. Inanother variation of this embodiment, the 3D image data comprises 3Dpoint cloud data and the one or more 3D image features includes a coloror a color gradation corresponding an object presented within the secondFOV. In another variation of this embodiment, the memory further storesinstructions that, when executed, cause the processor to identify theone or more 3D image features by identifying the one or more 3D imagefeatures that are positioned within a predetermined distance range awayfrom the 3D imaging apparatus. In some variations, the memory furtherstores instructions that, when executed, cause the processor to enhancethe 2D image data by filtering the at least one or more 2D imagefeatures such that the processing the enhanced 2D image data excludesprocessing of image data associated with the at least one or more 2Dimage features. In some variations, the memory further storesinstructions that, when executed, cause the processor to enhance the 2Dimage data by filtering the at least one or more 2D image features suchthat the processing the enhanced 2D image data is limited to processingof image data associated with the at least one or more 2D imagefeatures. In some variations, the system further includes a stationarybarcode reader configured to be positioned within a workstation andoperated by the operator, wherein the predetermined distance range awayfrom the 3D imaging apparatus extends from the 3D imaging apparatus toan edge of the workstation proximate the operator. In some variations,the system further includes a bi-optic barcode reader having aproduct-scanning region, wherein the predetermined distance range awayfrom the 3D imaging apparatus extends from the 3D imaging apparatus to adistal boundary of the product-scanning region.

In another variation of this embodiment, the one or more 3D imagefeatures includes at least one of (i) at least a portion of anoperator's hand, and (ii) an object grasped by the operator's hand.

In another variation of this embodiment, the memory further storesinstructions that, when executed, cause the processor to: process theenhanced 2D image data by identifying the action performed by theoperator; and responsive to the action performed by the operator beingidentified as one of presenting an object within a product-scanningregion and presenting the object proximate to the product-scanningregion, and further responsive to no barcode being detected within atleast one of the 2D image data and the enhanced 2D image data, generatean alert suitable for signaling a potential theft event.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying figures, where like reference numerals refer toidentical or functionally similar elements throughout the separateviews, together with the detailed description below, are incorporated inand form part of the specification, and serve to further illustrateembodiments of concepts that include the claimed invention, and explainvarious principles and advantages of those embodiments.

FIG. 1 is a perspective view of an example imaging system, implementedin an example point-of-sale (POS) system, having a bi-optical (alsoreferred to as “bi-optic”) symbology reader having a two-dimensional(2D) imaging apparatus and a three-dimensional (3D) imaging apparatus.

FIG. 2 is a perspective view of another example imaging system,implemented in an example POS system, having bi-optic symbology readerwith a 2D imaging apparatus and 3D imaging apparatus.

FIG. 3 is a perspective view of another example imaging system,implemented in an example POS system, having bi-optic symbology readerwith a 2D imaging apparatus and 3D imaging apparatus.

FIGS. 4A and 4B are a perspective view and a cross-sectional view,respectively, of another example imaging system, implemented in anexample POS system, having bi-optic symbology reader showing an internal3D imaging apparatus and showing a field of view of that 3D imagingapparatus.

FIG. 5 is a perspective view of another example imaging system,implemented in an example POS system, having bi-optic symbology readerhaving an associated 2D imaging apparatus within a reader housing and anassociated 3D imaging apparatus external to the reader housing.

FIG. 6 is a perspective view of another example imaging having ahandheld symbology reader having a 2D imaging apparatus and 3D imagingapparatus.

FIG. 7 illustrates a block diagram of an example logic circuit of ascanning station and remote server for implementing example methodsand/or operations described herein including those of the imagingsystems of FIGS. 1-6.

FIG. 8 is flowchart of an example process as may be implemented by thelogic circuit of FIG. 7, for implementing example methods and/oroperations described herein including techniques for barcode scanningusing captured 2D and 3D images.

FIG. 9A illustrates a 2D image captured by a 2D imaging apparatus of afirst field of view.

FIG. 9B illustrates a portion of a 3D image captured by a 3D imagingapparatus of a second field of view.

FIG. 9C illustrates an enhanced 2D image generated by extract imagefeatures associated with the 3D image.

FIG. 10 is flowchart of an example process as may be implemented by thelogic circuit of FIG. 7, for implementing example methods and/oroperations described herein including techniques for processing datausing captured 2D and 3D images.

FIG. 11A illustrates a 3D image captured by a 3D imaging apparatus of afirst field of view.

FIG. 11B illustrates a portion of a 2D image captured by a 2D imagingapparatus of a second field of view.

FIG. 11C illustrates an enhanced 3D image generated by extract imagefeatures associated with the 2D image.

FIG. 12 is flowchart of an example process as may be implemented by thelogic circuit of FIG. 7, for implementing example methods and/oroperations described herein including techniques for identifying aproper scanning and improper scanning using captured 2D images and 3Dimages.

FIG. 13 is flowchart of an example process as may be implemented by thelogic circuit of FIG. 7, for implementing example methods and/oroperations described herein including techniques for performing facialrecognition using captured 3D images.

FIG. 14A illustrates a 3D image captured by a 3D imaging apparatus of afield of view and showing a face at a first distance from the 3D imagingapparatus in a scanning station.

FIG. 14B illustrates a 3D image captured by the 3D imaging apparatus ofthe field of view and showing a face at a second distance from the 3Dimaging apparatus in a scanning station.

FIG. 15 is a flowchart of an example process as may be implemented bythe logic circuit of FIG. 7, for implementing example methods and/oroperations described herein including techniques for performing facialrecognition using captured 2D images.

FIG. 16 is a flowchart of an example process as may be implemented bythe logic circuit of FIG. 7, for implementing example methods and/oroperations described herein including techniques for performing facialrecognition using captured 3D images.

FIG. 17 is flowchart of another example process as may be implemented bya logic circuit of FIG. 18 in machine vision system, for implementingexample methods and/or operations described herein including techniquesfor performing facial recognition using captured 3D images.

FIG. 18 illustrates a block diagram of an example machine vision systemfor implementing example methods and/or operations described hereinincluding machine vision analysis using captured 2D and 3D images.

Skilled artisans will appreciate that elements in the figures areillustrated for simplicity and clarity and have not necessarily beendrawn to scale. For example, the dimensions of some of the elements inthe figures may be exaggerated relative to other elements to help toimprove understanding of embodiments of the present invention.

The apparatus and method components have been represented whereappropriate by conventional symbols in the drawings, showing only thosespecific details that are pertinent to understanding the embodiments ofthe present invention so as not to obscure the disclosure with detailsthat will be readily apparent to those of ordinary skill in the arthaving the benefit of the description herein.

DETAILED DESCRIPTION

FIG. 1 illustrates a perspective view of an example imaging systemcapable of implementing operations of the example methods describedherein, as may be represented by the flowcharts of the drawings thataccompany this description. In the illustrated example, an imagingsystem 100 is in the form of a point-of-sale (POS) system, having aworkstation 102 with a counter 104, a bi-optical (also referred to as“bi-optic”) symbology reader 106 (that may be used in the theftdetection and prevention, facial recognition, object recognition, andauthentication systems and methods described herein), and a first camera107 and a second camera 109, each at least partially positioned within ahousing of the symbology reader 106, also referred to herein as abarcode reader. In examples herein, the symbology reader 106 is referredto as a barcode reader.

In examples herein, the cameras 107 and 109 (as well as other camerasdescribed in other examples including those of FIGS. 2-4) may bereferred to as image acquisition assemblies and may be implemented as acolor camera, monochromatic camera, or other camera configured to obtainimages of an object. In the illustrated example, the camera 107 iswithin a vertically extending, upper housing 114 (also referred to as anupper portion or tower portion) of the barcode reader 106, and thecamera 109 is within a horizontally extending, lower housing 112 (alsoreferred to as a lower portion or platter portion). The upper housing114 is characterized by a horizontally extending field of view for thecamera 107. The lower housing 112 is characterized by a verticallyextending field of view for the camera 109. In some examples, the upperhousing 114 and lower housing 112 may each have two or more fields ofview, for example, if the respective cameras are configured (e.g.,through angle reflectors) to have different fields of view extendingfrom different pixel regions of the camera sensor.

In some examples, the lower housing 112 includes a weigh platter 111, aspart of a weigh platter assembly that will generally include the weighplatter 111 and an scale configured to measure the weight of an objectplaced on an example surface of the top portion 116, or some portionthereof. The surface of that top portion 116 extends in a firsttransverse plane and is generally or substantially parallel to anexample top surface 124 of the workstation 102 that at least partiallysurrounds the weigh platter 111.

The weigh platter 111 may be part of an off-platter detection assembly126 that includes an example light emission assembly 128 and an examplelight detection assembly 130. In an example, a light source (not shownfor clarity of illustration) of the light emission assembly 128 iscontrolled to emit one or more pulses of light, and a light sensor (notshown for clarity of illustration) of the light detection assembly 130captures light and the off-platter detection assembly 126 can processlight detection information to detect when a portion of an item, object,etc. is not resting on or is extending over an edge of the weigh platter111 as an off-platter weigh condition. For simplicity, only a singlelight emission assembly 128 and only a single light detection assembly130 are described herein, however, it will be understood thatoff-platter detection assembly 126 can also include any number and/ortype(s) of light emission assemblies, and any number and/or type(s)light detection assemblies may be implemented to detect off-platterweigh condition. Other example symbology readers with off-platterdetection discussed in reference to FIGS. 5-7.

In the illustrated example of FIG. 1, one of the cameras 107, 109 is a2D camera and the other of the cameras is a 3D camera. That is, invarious embodiments, the upper housing 114 may include a 2D camera or a3D camera, while the lower housing 112 may include a 3D camera or a 2Dcamera.

While the cameras 107 and 109 are shown in an example configuration eachin a different housing portion, the imaging systems herein may includeany number of imagers housed in any number of different devices. FIG. 2,for example, illustrates an imaging system 100′ having similar features(and like reference numerals) to imaging system 100 of FIG. 1, but inwhich a camera assembly 207 includes both a 2D camera 207A and a 3Dcamera 207B, each within an upper housing 114, without a separate camerain the lower housing 112. FIG. 3 illustrates another imaging system 100″having similar features (and like reference numerals) to imaging system100 of FIG. 1, but with a camera assembly 307 that includes a 2D camera307A and a 3D camera 307B, each with a lower housing 114, without aseparate camera in the upper housing 114.

Further, while FIGS. 1-3 illustrate example bi-optic barcode readers 106as the imager (each figure showing a different camera configuration), inother examples, the imager may be a handheld device, such as a handheldbarcode reader, or a fixed imager, such as barcode reader held in placein a base and operated within what is termed a “presentation mode.” Anexample in shown in FIG. 6, discussed further below.

Returning to FIG. 1, the lower housing 112 may be referred to as aplatter or first housing portion and the upper housing 114 may bereferred to as a tower, raised, or a second housing portion. The lowerhousing 112 includes a top portion 116 with a first opticallytransmissive window 118 positioned therein along a generally horizontalplane relative to the overall configuration and placement of the barcodereader 106. The window 118 coincides with a field of view(s) of thecamera 109 in the example of FIG. 1 and coincides with a field ofview(s) of the camera assembly 307 in the example of FIG. 3. In someexamples, the top portion 116 includes a removable or a non-removableplatter (e.g., a weighing platter). The upper housing 114 includes asecond optically transmissive window 120 positioned there along agenerally vertical plane and coinciding with a field of view(s) of thecamera 107.

In the illustrated example of FIG. 1, as well as FIGS. 2-4, the barcodereader 106 captures images of an object, in particular a product 122,such as for example a box, scanned by a user 108, e.g., a customer orsales clerk. In some implementations, the barcode reader 106 capturesthese images of the product 122 through one of the first and secondoptically transmissive windows 118, 120. For example, image capture maybe done by positioning the product 122 within the fields of view FOV ofthe digital imaging sensor(s) housed inside the barcode reader 106. Thebarcode reader 106 captures images through these windows 118, 120 suchthat a barcode 124 associated with the product 122 is digitally readthrough at least one of the first and second optically transmissivewindows 118, 120. Further in some examples, a graphic 126 on the productmay also be digitally analyzed through images captured through thesewindows 118, 120.

In example configurations of FIG. 1, 2D images of the product 122 arecaptured by the camera 107 over a field of view (FOV) of the camera 107,each 2D image being of environment appearing within that FOV. That is,in an example, the camera 107 is a 2D camera that captures 2D images andgenerates 2D image data that can be processed, for example, to verifythat the product 122 scanned matches the barcode 124 and/or image datacan be used to populate a database.

In example configurations of FIG. 1, 3D images of the product 122 arecaptured by the camera 109 over FOV of the camera 109, each 2D imagebeing of environment appearing within that FOV. That is, in an example,the camera 109 is a 3D camera that captures 3D images and generates 3Dimage data that can be processed, for example, to verify that theproduct 122 scanned matches the barcode 124 and/or image data can beused to populate a database. In other examples, camera 107 may be a 3Dcamera and camera 109 may be a 2D camera. While in FIG. 2, the camera207 includes a 2D camera 207A and a 3D camera 207B within the upperhousing 114, and in FIG. 3, the camera 307 includes a 2D camera 307A anda 3D camera 307B within a lower housing 112.

To implement operations of the example object detection techniquesherein, including those of FIGS. 8-16, images captured by 2D cameras andby the 3D cameras in the examples herein may be used for identifying anobject (such as the product 122), such as through determining a firstobject identification data using images captured of the object anddetermining a second object identification data using a symbology on theobject (such as, the barcode 124), and comparing the two identificationdata. In some examples, the images captured by the 2D camera(s) and 3Dcamera(s) are used to identify 2D image features and/or 3D imagefeatures, respectively, for controlling parameters associated withoperation of a symbology reader. In some examples, the images capturedby the 2D camera(s) and 3D camera(s) are used to identify facial datafor controlling parameters associated with operation of a symbologyreader or for authenticating operation of a symbology reader. Bycapturing and processing 2D image data and 3D image data, symbologyreaders in various examples are able to process enhanced 2D or 3D imagedata to train an object recognition model with the enhanced 3D imagedata, to recognize an object within enhanced image data, to identify anaction performed by a user of the barcode reader, and/or to change atleast one parameter associated with the symbology reader, whether thatparameter is associated with the 3D imaging apparatus, the 2D imagingapparatus, or other system associated with the symbology reader.

In the illustrated example of FIG. 1, the imaging system 100 includes aremote server 130 communicatively coupled to the barcode reader 106through a wired or wireless communication link. In some examples, theremote server 130 is communicatively coupled to a plurality of imagingsystems 100 positioned at checkout area of a facility, for example. Insome examples, the remote server 130 is implemented as an inventorymanagement server that generates and compares object identificationdata. In some examples, the remote server 130 is accessible by a managerfor monitoring operation and improper product scanning by the imagingsystem 100.

FIG. 4A illustrates another perspective view an example imaging system140 having a symbology reader 142, with a 3D imaging apparatus 144, inthis example within an tower portion 146 of the symbology reader 142. Aplatter portion 148 is similarly shown. A horizontally extending FOV 150for the 3D imaging apparatus 144 is shown. The 3D imaging apparatus 144,as with the other 3D imaging apparatuses in other examples herein, isable to capture 3D images of an environment within the FOV 150 andfurther determine the position of one or more objects in that field ofview. Example planes 152, 154, and 156 are shown each representing adifferent distance from the 3D imaging apparatus 144. Thus, objects andobject distances may be identified in 3D images and stored in 3D imagedata generated by the 3D imaging apparatus 144. In particular, asdiscussed further herein, the object type (e.g., a product to bescanned, an operator hand, or a human face) may be identified from 3Dimage data and the location of that object may be determined from tocontrol operation of the symbology reader 142. For example, in someexamples, the symbology reader 142 may be controlled to capture a 2Dimage for barcode reading only when an object of a certain type has beenidentified or when an object of a certain type is identified as beingwithin a distance of the symbology reader 142, such as between the 3Dimaging apparatus 144 and the distance set forth by plane 154. Objectsfurther away than plane 154 (such as objects at plane 156) may beignored and not scanned, for example. Similarly, objects closer thanplane 152 may be ignored or may be scanned using a differentillumination assembly or by adjusting a focusing plane of a 2D imagingapparatus. In some examples, the distance of objects appearing within anenvironment may be used to segment out certain objects from the 3D imagedata to increase scanning accuracy or to generate training images for anobject recognition model. Other examples are described.

FIG. 4B is a cross-sectional view of the imaging system 140 showing thesymbology reader 142 having an off-platter detection assembly 158, whichmay represent two off-platter detection assemblies one on each outerlongitudinal edge of the imaging system 140. To provide off-platterdetection, a light emission assembly 160 can be provided coupled to acontroller 162 of the symbology reader 140, where the controller 162 canbe configured to operate the off-platter detection assembly 158 invarious modes. For example, in a first mode of operation, controller 162can send an instruction to light emission assembly 160 to emitcollimated light beam 164 continuously, which would provide anindication of an off-platter event any time one of a light diffusingbarrier 166 is not illuminated. In a second mode of operation, thecontroller 162 can also be operatively coupled to weigh platter 168 andsend an instruction to the light emission assembly 160 not to emitcollimated light beam 164 until controller 162 detects an object on theweigh platter 168 and the measured weight of the object has settled.Once the measured weight has settled (after a positive dwell period),the controller 162 can send an instruction to light emission assembly160 to emit collimated light beam 164 to allow a user to determine ifthere is an off-platter event and to stop emitting collimated light beam164 once the controller 162 has detected removal of the object from theweigh platter 168. This mode of operation conserves energy and preventslight diffusing barrier 166 from continuously illuminating andde-illuminating every time a non-weighing object is passed over theweigh platter 168 to be scanned by the symbology reader 140. In a thirdmode of operation, the controller 162 is again operatively coupled tothe weigh platter 168. However, in this mode of operation the controller162 can send an instruction to light emission assembly 160 to blinkcollimated light beam 164 once the controller 162 detects an objectplaced on the weigh platter 168 to provide an alert to and draw theattention of the user that light diffusing barrier 166 should bechecked. The controller 162 can also generate a notification sound oralert to reminder the user to check light diffusing barrier 166. Oncethe measured weight of the object has settled (after a positive dwellperiod) the controller 162 then sends an instruction to the lightemission assembly 160 to stop blinking collimated light beam 164 and toemit collimated light beam 164 continuously.

In some examples, the collimated light beam 164 is emitted by a lightsource 170. The light emission assembly 160 may also include an aperture172, which could be formed in a wall or protrusion of a housing or couldbe formed through another wall or structure that is part of weighplatter 168, positioned in front of the light source 170 to focuscollimated light beam 164 into a narrow beam along a lateral edge of theweigh platter 168. A lens 174 can also be positioned in front ofaperture 172 to increase the intensity of the collimated light beam 164.

As shown, the 3D imaging apparatus 144 has the FOV 150 that extendsalong weigh platter 168 and encompasses light diffusing barrier 166 andis in communication with the controller 162. In this example, thecontroller 162 is configured to receive 3D images from the 3D camera 144and 2D images from a 2D camera 176 in the tower portion 146 and/or a 2Dcamera 178 in the platter portion 148. In some implementations, only oneof the 2D cameras may be provided. The controller 162 may be configuredto receive images from the cameras 144 and 176/178 and, based on theimages received, perform various operations described herein, such as,for example, adjusting one or more operating parameters of the symbologyreader 140. The controller 162 may additional determine if lightdiffusing barrier 166 appears to be illuminated or appears not to beilluminated, from 3D images captured by the camera 144 or from 2D imagescaptured by the camera 176. If the controller 162 determines that lightdiffusing barrier 166 appears to be illuminated, the controller 162 willallow the measured weight of an object on the weigh platter 168 to berecorded by a host system operatively coupled to the controller 162. Ifthe controller 162 determines that light diffusing barrier 166 appearsnot to be illuminated, the controller 162 will prevent the measuredweight of the object from being recorded by the host system and/or canprovide an alert to the user that there is on off-platter event.

FIG. 5 illustrates an alternative implementation of the imaging system140 of FIGS. 4A and 4B, labeled 140′ and using like reference numeralsto like elements. In contrast to the imaging system 140 in which the 3Dimaging apparatus 144 was internal to the symbology reader 142, in theimaging system 140′ 3D imaging apparatus 144′ is still associated withthe symbology reader 142 but is external to thereto and positionedoverhead with a vertical, downward extending FOV 150′. In the exampleshown, the 3D imaging assembly 144′ is within an overhead camera systemshown located in a gooseneck post 180 that extends from the back ofsymbology reader housing. However, the overhead camera system with the3D imaging assembly 144′ could be located anywhere above the housing ofthe symbology reader 142. For example, the overhead camera system couldbe located in a ceiling above symbology reader 142 and looking down onthe symbology reader 142, such as security cameras that are used withsome point-of-sale systems. In some examples, a single overhead camerasystem with one or more 3D imaging apparatuses may look down on one ormore imaging systems.

FIG. 6 illustrates another example imaging system capable ofimplementing operations of the example methods described herein, as maybe represented by the flowcharts of the drawings that accompany thisdescription. In the illustrated example, an imaging system 200 includesa symbology reader in the form of a handheld or presentation barcodereader 202 that may be used in the object imaging and other methodsdescribed herein. In the illustrated example, the barcode reader 202 mayinclude a handheld reader 204 and a stationary cradle 206 mounted to aworkstation surface 208. In the illustrated example, the handheld reader204 rests in the stationary cradle to establish a hands-free scanningmode, also termed a presentation mode, for scanning objects. Thehandheld reader 204 therefore operates as an imaging reader, with ascanning window 210 in the housing of the handheld reader 204, behindwhich is a camera assembly 212 and optionally an illumination assembly211 (which may represent one or more different illumination sources,such as a direct illumination source and a diffuse illumination sourcefor capturing 2D and/or 3D images). In the hands-free scanning mode, thehandheld reader 204 defines a FOV 214 having a central axis 215. Inaccordance with the techniques herein, the handheld reader 204 capturesimages of an object for identification and imaging within the FOV 214. Atrigger 216 may be used to initiate a hands-free scanning mode, in someexamples. In some examples, the hands-free scanning made is initiated byplacement of the reader 204 into the cradle 206.

In the illustrated example, the camera assembly 212 includes a 2D camera212A for capturing 2D images of an object in the FOV 314 and generating2D image data. The camera assembly 212 further includes a 3D camera 212Bfor capturing 3D images of an object in the FOV 314 and generating 3Dimage data. While the FOV 314 may be for each of the cameras 212A and212B, in other examples, each of the cameras 212A and 212B may have adifferent FOV, either partially overlapping or non-overlapping. In someexamples, the FOV 314 may include a first portion corresponding to the2D camera 212A and a second portion corresponding to 3D camera 212B.

FIG. 7 illustrates an example system where embodiments of the presentinvention may be implemented. In the present example, the environment isprovided in the form of a facility having one or more scanning locations300 corresponding to an imaging system, such as the imaging systems 100,100′, 100″, 140, 140″ and 200 of 1, 2, 3, 4A, 4B, 5, and 6,respectively, where various objects may be scanned for completing apurchase of an object, for improper object detection to override animproper purchase attempt, and other purposes herein.

In an example, the scanning location 300 is a point-of-sale location andincludes a scanning station 302 having a symbology reader 304, such as abi-optic barcode reader like barcode reader 106 in FIGS. 1-5 and thehandheld barcode reader 202 in FIG. 6. The symbology reader 304 mayinclude a scanner, such as a barcode scanner, and any additional type ofsymbology reader, such as an RFID tag reader. In the example of FIG. 1,the symbology reader 304 is also described as a reader 304, forconvenience sake, although any type of symbology reader is meant to beincluded.

In another example, the scanning location is a machine vision locationand the scanning station 302 is a machine vision system having a 2Dimaging apparatus and a 3D imaging apparatus and configured to performprocesses such as those described in reference to FIG. 11 and elsewhereherein.

The reader 304 includes an imaging device 306 (e.g., an imaging assemblyin the form of a camera assembly 306 or other photodetection device) andone or more sensors 308. The camera assembly 306 includes a 2D camera310 and a 3D camera 312 for capturing respective images and generatingrespective image data, in accordance with the techniques describedherein. As discussed in various examples herein, the 2D camera 310 and3D camera 312 are associated with the symbology reader 304 and may bewithin a housing thereof or external to a housing but coupled to thesymbology reader 304 through a wired or wireless communication link. Insome examples, the reader 304 may be a barcode image scanner capable ofscanning a 1D barcode, QR code, 3D barcode, or other symbology as theindicia 314, as well as capturing images of an object 316 itself. In theillustrated example, the scanning station 304 includes sensors 308,which may include an RFID transponder for capturing indicia data is theform of an electromagnetic signal captured from the indicia 314 when theindicia 314 is an RFID tag, instead of an visual indicia, such as abarcode.

The reader 304 further includes an image processor 318 and an indiciadecoder 320. In some examples, the image processor 318 is a 2D imagedata processor and a 3D image data processor, capable of processingcolor image data, point cloud image data, and then the like of anobject.

The image processor 318 may be configured to analyze captured images ofthe object 316 and perform preliminary image processing, e.g., before 2Dand 3D images and image data is sent to a remote server 350 via anetwork 324. The reader 304 includes a network interface 326 thatrepresents any suitable type of communication interface(s) (e.g., wiredand/or wireless interfaces) configured to operate in accordance with anysuitable protocol(s) for communicating over the network 324.

In the example of a bi-optic barcode reader, the symbology reader 304includes an off-platter detection assembly 321 that may include a lightemission assembly and light detection assembly, different than thecamera assembly 306.

In the illustrated example, the reader 304 includes a processingplatform 323 having a processor 328 such as, for example, one or moremicroprocessors, controllers, and/or any suitable type of processor. Theprocessing platform 323 further includes a memory 330 (e.g., volatilememory, non-volatile memory) accessible by the processor 328 (e.g., viaa memory controller). The example processor 328 interacts with thememory 330 to obtain, for example, machine-readable instructions storedin the memory 330 corresponding to, for example, the operationsrepresented by the flowcharts of this disclosure, including those ofFIGS. 8, 10, 12, 13, 15, and 16. Additionally or alternatively,machine-readable instructions corresponding to the example operationsdescribed herein may be stored on one or more removable media (e.g., acompact disc, a digital versatile disc, removable flash memory, etc.)that may be coupled to the reader 304 to provide access to themachine-readable instructions stored thereon.

The process 328 may be a programmable processor, a programmablecontroller, a graphics processing unit (GPU), a digital signal processor(DSP), etc. Alternatively, an application specific integrated circuit(ASIC), a programmable logic device (PLD), a field programmable gatearray (FPGA), a field programmable logic device (FPLD), a logic circuit,etc. may be structured or configured to implement the processor 328.Example memory 330 include any number or type(s) of non-transitorycomputer-readable storage medium or disk, such as a hard disk drive(HDD), an optical storage drive, a solid-state storage device, asolid-state drive (SSD), a read-only memory (ROM), a random-accessmemory (RAM), a compact disc (CD), a compact disc read-only memory(CD-ROM), a digital versatile disk (DVD), a Blu-ray disk, a cache, aflash memory, or any other storage device or storage disk in whichinformation may be stored for any duration (e.g., permanently, for anextended time period, for a brief instance, for temporarily buffering,for caching of the information, etc.).

The reader 304 of FIG. 7 also includes input/output (I/O) interfaces 332to enable receipt of user input and communication of output data to theuser, an input device 334 for receiving input from the user, and adisplay 336 device for displaying data, alarms, and other indications toa user. In the illustrated example, the I/O interfaces 332 are part ofthe processing platform 323 along with the network interface 326.Although FIG. 7 depicts the I/O interfaces 332 as a single block, theI/O interfaces 332 may include a number of different types of I/Ocircuits or components that enable the processor 328 to communicate withperipheral I/O devices. Example interfaces 332 include an Ethernetinterface, a universal serial bus (USB), a Bluetooth® interface, a nearfield communication (NFC) interface, and/or a PCI Express interface. Theperipheral I/O devices may be any desired type of I/O device such as akeyboard, a display (a liquid crystal display (LCD), a cathode ray tube(CRT) display, a light emitting diode (LED) display, an organic lightemitting diode (OLED) display, an in-place switching (IPS) display, atouch screen, etc.), a navigation device (a mouse, a trackball, acapacitive touch pad, a joystick, etc.), a speaker, a microphone, aprinter, a button, a communication interface, an antenna, etc.

To perform indicia identification, in some examples, the image processor318 is configured to identify the indicia 314 captured in a 2D image,e.g., by performing edge detection and/or pattern recognition, and theindicia decoder 320 decodes the indicia and generates objectidentification data corresponding to the indicia 314.

In some examples, the image processor 318 is further configured tocapture images of the object 316 and determine other objectidentification data based on features in the 2D image data, the 3D imagedata, or a combination thereof.

For example, in some examples, the image processor 318 identifies imagefeatures of the object 316 captured by in 2D image data. Example imagefeatures include the peripheral shape of the object, the approximatesize of the object, a size of the packaging portion of the object, asize of a product within a packaging (e.g., in the case of a packagedmeat or produce), a relative size difference between a size of theproduct and a size of the packaging, a color of the object, packaging,and/or good, images on the object, text on the object, point-of-salelane and store ID from where the item was scanned, shape of product,weight of product, variety of the product especially for fruits, andfreshness of the product.

The image features identified by the image processor 318 can becategorized into image features derived from 2D image data captured bythe 2D camera 310 and image features derived from 3D image data capturedby the 3D camera 312. The image processor 318 may send these imagefeatures within the image scan data to the remote server 350.

In some examples, the image processor 318 is configured to identify oneor more 3D image features in the captured 3D image data and identifyingone or more 2D image features in the captured 2D image data. The 3Dimage data may be point cloud data and the 3D image features may includecolor data for objects, color gradient data for objects, shape data forobjects, dimension data for objects, distance data for objects relativeto other objects or relative to the symbology reader or relative to the3D camera, or other 3D features. The 2D image features may be shape datafor objects, dimension data for objects, graphics on objects, labels onobjects, text on objects, or other 2D features.

While in some examples, the 2D and 3D image features are identified inthe image processor 318, in other examples, 2D image featuredetermination and 3D image feature determination are performed in animage features manage 338 having stored 2D images features 340 andstored 3D image features 342.

Whether the 2D image features and/or 3D image features are determined inthe image processor 318 or in the image features manager 338, in variousexamples, the reader 304 is configured to enhance decoding of theindicia 314 captured in 2D image data using captured 3D image data. Invarious examples, the reader 304 is configured to enhance 2D image dataand/or 2D image data analysis with 3D image data. In various examples,the reader 304 is configured to enhance 3D image data and/or 3D imagedata analysis with 2D image data. In various examples, the reader 304 isconfigured to enhance theft detection and prevention, facialrecognition, object recognition, and/or authentication processes usingboth the 2D image data and the 3D image data. In various examples, thereader 304 is configured to perform these processes (i.e., theftdetection and prevention, facial recognition, object recognition, and/orauthentication) using only 3D image data captured by the reader 304. Toperform these and other processes in accordance with the techniquesherein, the image features manager 338 may be configured to performprocesses as detailed in FIGS. 8-16, described further hereinbelow.Furthermore, in some examples, one or more of these processes may beperformed at the remote server 350, which may include an image featuresmanager 351 configured to perform one or more of the processes describedherein, such as those detailed in FIGS. 8-16.

In an example, the remote server 350 is an image processing and objectidentification server configured to receive 2D images (2D image data)and 3D images (3D image data) (and optionally other image scan data,such as decoded indicia data, physical features, etc.) from the scanningstation 302 and perform object identification such as objectidentification and improper object detection and other techniquesdescribed herein, including at least some of the processes described inreference to FIGS. 8-16.

The remote server 350 may implement enterprise service software that mayinclude, for example, RESTful (representational state transfer) APIservices, message queuing service, and event services that may beprovided by various platforms or specifications, such as the J2EEspecification implemented by any one of the Oracle WebLogic Serverplatform, the JBoss platform, or the IBM WebSphere platform, etc. Othertechnologies or platforms, such as Ruby on Rails, Microsoft .NET, orsimilar may also be used.

The remote server 350 includes an example logic circuit in the formprocessing platform 352 capable of, for example, implementing operationsof the example methods described herein include. The processing platform352 includes a processor 354 such as, for example, one or moremicroprocessors, controllers, and/or any suitable type of processor. Theexample processing platform 352 includes memory (e.g., volatile memory,non-volatile memory) 356 accessible by the processor 354 (e.g., via amemory controller). The example processor 354 interacts with the memory356 to obtain, for example, machine-readable instructions stored in thememory 356 corresponding to, for example, the operations represented bythe flowcharts of this disclosure. Additionally or alternatively,machine-readable instructions corresponding to the example operationsdescribed herein may be stored on one or more removable media (e.g., acompact disc, a digital versatile disc, removable flash memory, etc.)that may be coupled to the processing platform 352 to provide access tothe machine-readable instructions stored thereon.

The example processing platform 352 also includes a network interface358 to enable communication with other machines via, for example, one ormore networks, including the scanning station 302. The example networkinterface 358 includes any suitable type of communication interface(s)(e.g., wired and/or wireless interfaces) configured to operate inaccordance with any suitable protocol(s), as well the network interface326. The example, processing platform 352 also includes input/output(I/O) interfaces 360 to enable receipt of user input and communicationof output data to the user, in a similar manner to that of I/O interface332.

In some examples, the remote server 350 may be, or may contain, aclassification server. As such, in the illustrated example, the remoteserver 350 includes, in a neural network framework 362 configured todevelop a trained neural network 364 and to use that trained neuralnetwork to receive 2D and 3D images captured by the symbology reader304, identify objects within the received images, and classify thoseobjects. The neural network framework 362, for example, may beconfigured as a convolutional neural network employing a multiple layerclassifier to assess identified image features and generate classifiersfor the trained neural network 364.

The trained neural network 364 may be trained to classify objects inreceived 2D and/or 3D image data by object type, scanning surface of theobject, whether the object is a display screen such as a mobile devicedisplay, reflectivity of the object, and/or type of indicia on object.By way of example, the trained neural network 364 may be trained using2D images, 2D image data enhanced with 3D image data, 3D image data,and/or 3D image data enhanced with 2D image data.

In these ways, through the framework 362 and the trained neural network364, in various examples, the present techniques deploy one or moretrained prediction models to assess received 2D and/or 3D images of anobject (with or without indicia) and classify those images to determinean object and object classification, for identifying contextualconfiguration settings to use in operating the scanning station 302 orother imaging devices.

From the determined classifications, in various examples, the presenttechniques use an object's classification to determine adjustments forconfiguration settings of the imaging scanner. That is, a predictionmodel is trained using the neural network framework k362, and as suchthat prediction model is referred to herein as a “neural network” or“trained neural network.” The neural network herein may be configured ina variety of ways. In some examples, the neural network may be a deepneural network and/or a convolutional neural network (CNN). In someexamples, the neural network may be a distributed and scalable neuralnetwork. The neural network may be customized in a variety of manners,including providing a specific top layer such as but not limited to alogistics regression top layer. A convolutional neural network can beconsidered as a neural network that contains sets of nodes with tiedparameters. A deep convolutional neural network can be considered ashaving a stacked structure with a plurality of layers. In examplesherein, the neural network is described as having multiple layers, i.e.,multiple stacked layers, however any suitable configuration of neuralnetwork may be used.

CNNs, for example, are a machine learning type of predictive model thatare particularly using for image recognition and classification. In theexemplary embodiments herein, for example, CNNs can operate on 2D or 3Dimages, where, for example, such images are represented as a matrix ofpixel values within the image scan data. As described, the neuralnetwork (e.g., the CNNs) can be used to determine one or moreclassifications for a given image by passing the image through theseries of computational operational layers. By training and utilizingtheses various layers, the CNN model can determine a probability that animage (or object(s) within an image) or physical image features belongsto a particular class. Trained CNN models can be persisted forrestoration and use, and refined by further training. Trained models canreside on any in-premise computer volatile or non-volatile storagemediums such as RAM, flash storage, hard disk or similar storage hostedon cloud servers.

In FIG. 8, a flowchart is shown of an example process 400 for symbologyreading using captured 2D images and captured 3D images as may beperformed by the scanning station 302 of FIG. 7 and any of the imagingsystems 100, 100′, 100″, 140, 140″ and 200 of 1, 2, 3, 4A, 4B, 5, and 6,respectively. At a block 402, a symbology reader of an imaging systemcaptures one or more 2D images through a 2D imaging apparatus of thesymbology reader. In particular, 2D images are captured over a firstenvironment within a FOV, where that first environment may be from ahorizontally extending FOV of an upper portion of a bi-optic reader,from a vertically extending FOV of a lower portion of a bi-optic reader,from a FOV of a handheld symbology reader, or other example. At a block404, a 3D imaging apparatus of the symbology reader captures one or more3D images of another environment of a second FOV corresponding to the 3Dimaging apparatus. The FOV of the 3D imaging apparatus may be from anupper portion or a lower portion of the bi-optic reader or from anexternally positioned 3D imaging apparatus associated with a symbologyreader, such as from a FOV extending downward overhead or from a sideview pointing toward the symbology reader. The FOV may be that of a 3Dimaging apparatus in a handheld symbology reader. In any event, in someexamples, the 2D imaging apparatus and the 3D imaging apparatus havedifferent FOVs that partially overlap. In some examples, the FOVs of the2D imaging apparatus and the 3D imaging apparatus are the same.

In some examples, multiple 2D imaging apparatuses/assemblies may be usedto capture 2D images, such as 2D imaging apparatuses in each of an upperportion and a lower portion of a symbology reader.

At a block 406, the imaging system analyzes 3D image data and identifies3D image features within the environment captured in the FOV of the 3Dimaging apparatus. For example, the 3D image data may be 3D point clouddata, and the 3D image features may be one or more geometric features ofan object presented within the FOV and/or color or a color gradationcorresponding an object within the FOV. In some examples, theidentifying 3D image features includes identifying the one or more 3Dimage features that are positioned within a predetermined distance rangeaway from the 3D imaging apparatus. In the imaging system 140 of FIG.4A, for example, 3D imaging apparatus 144 may be configured to detectone or more objects captured in a 3D image of the FOV 150, determine thedistance of the one or more objects, and determine if relevant objectsare beyond a determined maximum distance, such as the distance plane156. In the example, of a bi-optic barcode reader having aproduct-scanning region, the predetermined distance range away from the3D imaging apparatus may extend from the 3D imaging apparatus to adistal boundary of the product-scanning region, e.g., distance plane156. In some examples, the 3D image features include at least one of (i)at least a portion of an operator's hand, and (ii) an object grasped bythe operator's hand.

In the example of a stationary barcode reader configured to bepositioned within a workstation and operated by the operator, such as ahandheld barcode reader like that of FIG. 6, the predetermined distancerange away from the 3D imaging apparatus may extend from the 3D imagingapparatus to an edge of the workstation proximate the operator.

At a block 408, having analyzed the 3D image data and identifying 3Dimage features, the imaging system enhances the 2D image data bycorrelating the identified 3D image features with 2D image featuresidentified from 2D image data of the 2D images captured in theenvironment of the FOV of the 2D imaging apparatus. For example,enhancing the 2D image data may include filtering one or more 2D imagefeatures such that processing the enhanced 2D image data excludesprocessing of image data associated with the at least one or more 2Dimage features. In some examples, enhancing the 2D image data furtherincludes filtering the one or more 2D image features such thatprocessing the enhanced 2D image data is limited to processing of imagedata associated with the at least one or more 2D image features. Inthese ways, 3D image features can be used to improve the processing of2D images, removing the processing of 2D image features determined notbe valuable for processing and/or identifying 2D image features that arevaluable for processing.

To illustrate an example, FIG. 9A illustrates a 2D image of a firstenvironment of a FOV captured by a 2D imaging assembly of an imagingsystem, in this example from upper portion of the imaging system. The 2Dimage includes a barcode 450 associated with a target 452. The 2D imagecontains more than the target 452 and barcode 450. The 2D image includesa clerk 454 and background objects (e.g. chair 456, and person 458).

To enhance analysis of the 2D image in the example of FIG. 9A, the block406 uses 3D image data of a 3D image captured over a FOV for a 3Dimaging apparatus. For example, 3D image features corresponding to theobjects 452-458 may be determined, such as the distance of each of theobjects from a symbology reader. The FOV of the 3D image may be from anupper portion or a lower portion of a bi-optical reader or from anexternal, associated position, such as overhead. For example, at theblock 406, the distance of the target 452, the clerk 454, the chair 456,and the person 458 may be determined. Further, at the block 406 it maybe determined which of the objects are within an allowed scanningdistance for a symbology reader and which ones are not so that the latercan be removed. For illustration purposes, FIG. 9B shows an exampleportion of a 3D image where the chair 456 and the person 458 areidentified as being outside of an allowed scanning distance, which the3D imaging apparatus can determine from 3D point cloud data anddetermining 3D image features including geometric figures identify theshape of the object and distance of the object. In some examples, the 3Dimage features may be color or color gradation used to identifydifferent objects in the 3D image. In some examples, the 3D imagefeatures include at least a portion of an operator's hand and an objectgrasped by the operator's hand. In some examples, identified 3D imagefeatures may be sent a trained neural network, having one or moretrained object recognition models, for object identification. The objectrecognition models may be trained to identify different objects whetherwithin or outside of an allowed scanning distance.

With the distance data determined from the 3D image features, the 2Dimage of FIG. 9A may be converted into an enhanced 2D image as in FIG.9C, at the block 408, by, in an example, removing from the 2D imagesobjects that have distances outside of an allowed scanning distance,also referred to as a predetermined distance range of a product-scanningregion herein. For example, a predetermined distance range away from the3D imaging apparatus may extend from the 3D imaging apparatus to adistal boundary of the product-scanning region or to an edge of aworkstation proximate to the operator or to a diffuser of an off-platterdetection assembly.

For example, the enhanced 2D image of FIG. 9C has the chair 456 and theperson 458 removed. In some examples, the block 408 compares identified2D image features determined from 2D images and identified 3D imagefeatures determined from 3D images to identify objects for analysis.

The block 408 may enhance 2D image data such that certain image data(e.g., one or more objects) is excluded from further processing at block410 or such that image processing is limited to certain image data.

Various processing may be advantaged as a result of generating enhanced2D image data, including, as shown at block 410, decoding barcodescaptured within the enhanced 2D image data (such as faster, moreaccurate decoding of the barcode 450), train object recognition modelssuch as that of the trained neural network 364 using the enhanced 2Dimage data, perform object recognition using the enhanced 2D image data,identifying an action performed by an operator using the enhanced 2Dimage data, or change one or more parameters associated with operatingthe barcode reader.

For example, at the block 410, the action performed by the operator maybe identified as one of presenting an object within a product-scanningregion and presenting the object proximate to the product-scanningregion. With the enhanced 2D image data, the block 410 may, responsiveto no barcode being detected within at least one of the 2D image dataand the enhanced 2D image data, generate an alert suitable for signalinga potential theft event. In the example of FIG. 9A, if the 2D imagingapparatus determines that the target 452 is present in 2D image data andwithin an allowed scanning distance (e.g., a product-scanning region),as determined from 3D image features, but that the barcode 450 is notpresent, is partially covered or not visible, or fully covered or notvisible, and thus not fully decodable, then the block 410 may identifyan operator as attempting a product-scan, and the block 410 may generatean alarm signal or other indication suitable for signaling a potentialtheft event.

In some examples, the block 410, responsive to analyzing the 2D imagedata using the 3D image features may adjust one or more parametersassociated with symbology reader operation. Example features include anexposure time of the symbology reader, an illumination pulse duration ofthe symbology reader, a focus position of the symbology reader, animaging zoom level of the symbology reader, and an illumination sourceof the symbology reader. For example, wherein the position of the target452 is not detectable from 2D image data, by using 3D image features,the distance is determinable. Depending on that distance, the block 410may switch for a direct illumination source used for barcode reading toa diffuse illumination source use for direct part marking (DPM)symbology reading. In some examples, the block 410 may adjust thebrightness of an illumination assembly prior to capturing further 2Dimages. For example, if targets are close to the 3D imaging assemblythen a reduce illumination brightness may be used. Or, if objects suchas a facial data are identified in 3D images, then the illuminationbrightness may be reduced or the illumination assembly may be turnedoff.

One or more aspects of the processing operation of block 410 may beperformed at the scanning station 302 or at the server 350, for example.

In FIG. 10, a flowchart is shown of an example process 500 for symbologyreading using captured 2D images and captured 3D images as may beperformed by the scanning station 302 of FIG. 7 and any of the imagingsystems 100, 100′, 100″, 140, 140″ and 200 of 1, 2, 3, 4A, 4B, 5, and 6,respectively. At a block 502, similar to block 402, a symbology readerof an imaging system captures one or more 2D images through a 2D imagingapparatus of the symbology reader. In particular, 2D images are capturedover a first environment within a FOV. At a block 504, similar to block404, a 3D imaging apparatus associated with the symbology readercaptures one or more 3D images of another environment of a second FOVcorresponding to the 3D imaging apparatus.

At a block 506, the imaging system analyzes 2D image data and identifies2D image features within the environment captured in the FOV of the 3Dimaging apparatus. For example, the 2D image data may be monochromeimage data, grayscale image data, and polychrome image data. The 2Dimage features may be a barcode and one or more geometric features of anobject presented within the FOV of the 2D imaging apparatus. In someexamples, the 2D image features include at least one of (i) at least aportion of an operator's hand, and (ii) an object grasped by theoperator's hand, such as in the example of FIG. 9A the operator 454 andan operator's hand holding the target 452. In some examples, identifying2D image features includes identifying environmental features on the 2Dimage, where these environmental features are features in the imageoutside of an object presented within the first FOV, such as in theexample of FIG. 9A the chair 456 and the person 458. The block 506 mayconvert these environmental features into masking features configured tocover the identified environmental features in the 2D image and thenidentify these masking features as one or more 2D image features. Insome examples, the block 506 identifies 2D image features by identifyinga barcode for the object in the 2D image data and decoding the barcodeto generate barcode payload data. From there, the block 506 maydetermine object identification from the barcode payload data anddetermine, from the object identification, one or more 2D imagefeatures, such as the shape and/or dimensions of the object.

At a block 508, having analyzed the 2D image data and identifying 3Dimage features, the imaging system enhances the 3D image data bycorrelating the identified 2D image features with 3D image featuresidentified from 3D image data of the 3D image(s) captured in theenvironment of the FOV of the 3D imaging apparatus. For example,enhancing the 3D image data may include mapping a location of thebarcode from the 2D image data to the 3D image data. In another example,enhancing the 3D image data may include mapping at least a portion of apolychrome image data to the 3D image data based at least in part on theone or more geometric features of the object presented within the FOV ofthe 2D imaging apparatus. In some examples, enhancing the 3D image datamay include filtering one or more 3D image features such that processingthe enhanced 3D image data excludes processing of image data associatedwith the at least one or more 3D image features. In some examples,enhancing the 3D image data may include filtering one or more 3D imagefeatures such that processing the enhanced 3D image data is limited toprocessing of image data associated with the at least one or more 3Dimage features. For example, shape and dimension data may be determinedfrom a decoded barcode in the 2D image data and used to identify anobject in the 3D image data corresponding to that shape and dimension.In some examples, enhancing the 3D image data includes filtering the 3Dimage databased on a predetermined distance range away from the 3Dimaging apparatus. For example, for some handheld barcode readers, thepredetermined distance range away from the 3D imaging apparatus extendsfrom the 3D imaging apparatus to an edge of the workstation proximatethe operator. For some bi-optic readers, the predetermined distancerange away from the 3D imaging apparatus extends from the 3D imagingapparatus to a distal boundary of the product-scanning region. In theseways, 2D image features can be used to improve the processing of 3Dimages, removing the processing of 3D image features determined not bevaluable for processing and/or identifying 3D image features that arevaluable for processing.

FIG. 11A illustrates an example 3D image identifying a target 550 in anallowed scanning distance and operator 552 holding the target 550 intheir hand 554. A chair 556 and a person 558 are also identified in the3D image. FIG. 11B illustrates a filter mask developed from the 2D imagefeatures corresponding to the chair 556 and the person 558. FIG. 11Cillustrates enhanced 3D image data where the 2D image features of FIG.11B have been determined to be environmental features that are convertedto masking features, and the corresponding 3D image data correspondingto these features has been removed to form the enhanced 3D image of FIG.11C.

Various processing may be advantaged as a result, including, as shown atblock 510, training object recognition models such as those of trainedneural network 364 using enhanced 3D image data, performing objectrecognition using enhanced 3D image data, identifying an actionperformed by an operator, or and adjusting at least one parameterassociated with the 3D imaging apparatus. For example, the block 510 mayanalyze the enhanced 3D image data identifying the action performed bythe operator as one of presenting an object within a product-scanningregion and presenting the object proximate to the product-scanningregion. In response to no barcode being detected within the 2D imagedata, the process 510 may generate an alert suitable for signaling apotential theft event. One or more aspects of the processing operationof block 510 may be performed at the scanning station 302 or at theserver 350, for example.

In some examples, the block 510 may provide the enhanced 3D image datato train an object recognition model of a neural network framework. Forexample, the block 506 may identify a barcode in 2D image data anddetermine a barcode detection event timeframe. The block 508 maygenerate enhanced 3D images as a 3D image captured corresponding to thatbarcode detection event timeframe, and the block 510 may provide that 3Dimage to train an object recognition model. In this example, only 3Dimage data corresponding to a symbology decoding event is used to trainan object recognition model. Further, the 3D image used to train theobject recognition model may be further enhanced by removing objects togenerate an enhanced 3D image (e.g., FIG. 11C), which is used fortraining.

Further, in some examples, the block 510 adjusts one or more operatingparameters of the symbology reader, e.g., one or more operatingparameters of the 3D imaging apparatus such as the projected amount ofillumination of the 3D imaging apparatus, the projected direction ofillumination of the 3D imaging apparatus, the a source of illuminationfor the 3D imaging apparatus.

One or more aspects of the processing operation of block 510 may beperformed at the scanning station 302 or at the server 350, for example.

In FIG. 12, a flowchart is shown of an example process 600 for symbologyreading using captured 2D images and captured 3D images as may beperformed by the scanning station 302 of FIG. 7 and any of the imagingsystems 100, 100′, 100″, 140, 140″ and 200 of 1, 2, 3, 4A, 4B, 5, and 6,respectively. At a block 602, similar to block 402, a symbology readerof an imaging system captures one or more 2D images through a 2D imagingapparatus of the symbology reader. In particular, 2D images are capturedover a first environment within a FOV. At a block 604, similar to block404, a 3D imaging apparatus of the symbology reader captures one or more3D images of another environment of a second FOV corresponding to the 3Dimaging apparatus.

At a block 606, the imaging system determines first objectionidentification data using the captured 2D image data and second objectidentification data using the captured 3D image data. In some examples,both the first object identification data and the second objectidentification data are determined at a scanning station, e.g., at thesymbology reader 304. In some examples, one or both of these objectidentification data are determined at a remote server, e.g., the server350. For example, the first object identification data may be determinedat a symbology reader, and the second object identification data may bedetermined by using a trained neural network stored at that symbology orstored remotely at the server 350.

In the illustrated example, at block 608, the imaging system comparesthe first object identification to the second object identification anddetermines that a proper scanning of the object has occurred when thereis a match or determines that an improper scanning has occurred whenthere is no match. In some examples, in response to determining theproper scanning of the object, the block 608 may further include theimaging system (e.g., a POS, symbology reader, and/or remote server)processing a transaction log to include the data associated with theobject. In some examples, in response to determining the improperscanning of the object, the imaging system may generate an alertsuitable for signaling a potential theft event and/or process atransaction log to not include the data associated with the object.

The block 606 may be implemented in various ways. In an example,determining the first object identification of the object using the 2Dimage data includes identifying a barcode for the object in the 2D imagedata and decoding the barcode to generate barcode payload data, anddetermining the first object identification from the barcode payloaddata. In an example, determining the first object identification of theobject using the 2D image data includes providing the 2D image data to atrained object recognition model, such as the trained neural network364, and producing, using the trained object recognition model, thefirst object identification of the object. In an example, determiningthe second object identification of the object using the 3D image dataincludes providing the 3D image data to a trained object recognitionmodel and producing, using the trained object recognition model, thesecond object identification of the object.

In another example of the block 606, before determining the first objectidentification of the object using the 2D image data, the imaging systemcompares 3D image data to 2D image data, and removes environmentalfeatures outside of the object from the 2D image data based on the 3Dimage data. In another example, before determining the second objectidentification of the object using the 3D image data, the imaging systemcompares the 3D image data to the 2D image data and removesenvironmental features outside of the object from the 3D image databased on the 2D image data.

In yet another example of the block 606, determining the second objectidentification of the object using the 3D image data includes,determining one or more color features of the object from the 3D imagedata and from that determining the second object identification from theone or more color features. These color features may include the colorand or color gradient of an object. Such functionality may be used toidentify different produce for example by examining the color and/orcolor gradient of captured 3D image data of that produce.

In another example of the block 606, determining the second objectidentification of the object using the 3D image data includesdetermining one or more geometric features of the object from the 3Dimage data and determining the second object identification from the oneor more geometric features.

In another example, determining the first object identification of theobject using the 2D image data includes identifying a barcode for theobject in the 2D image data and decoding the barcode to generate barcodepayload data and determining the first object identification from thebarcode payload data. Where the 3D image data includes a point cloudhaving a plurality of data points, each data points having a distancevalue associated with a distance from the 3D imaging apparatus may bedetermined from one or more geometric features of the object. Forexample, the block 606 may determine that one or more geometric featuresof the object from the 3D image data are based on a first subset of the3D image data and not based on a second subset of the 3D image data. Thefirst subset of the 3D image data may be associated with a first subsetof the data points having the respective distance value associated withthe distance from the 3D imaging apparatus being within a predeterminedrange, and the second subset of the 3D image data may be associated witha second subset of the data points having the respective distance valueassociated with the distance from the 3D imaging apparatus being outsideof the predetermined range.

In this way, in some examples, 3D image features may be used with 2Dimage features to identify an object attempting to be scanned anddetermine whether the object scan attempt is within an allowed scanningdistance. In the example of FIG. 9A, the barcode 450 (as a first objectidentification data) may be identified in a 2D image and the location ofthe target 452 associated with the barcode 450 may be identified from 3Dimage data, along with the distance between the target 452 and thesymbology reader (as a second object identification data). If thatdistance is shorter than an allowed scanning distance, e.g., shorterthan the distance of plane 152 in the example of FIG. 4A or longer thanthe distance plane 156 (the region of the FOV between the planes 152 and156 defining an allowed scanning distance in this example), then theblock 608 comparing the first and second object identification data maydetermine that an improper scan attempt has been attempted and therebyblock decoding of the barcode 450 and/or blocking the inclusion of thetarget 452 and barcode 450 in a transaction log associated with thescanning station.

In another example, the block 604 may identify a scannable object usingthe 3D image data, and if the block 606 fails to determine an objectidentification of the object using the 2D image data, the block 608 maydetermine an improper scanning of the object and generating an alarmsignal. For example, a 2D image may be captured at the block 602, butthat 2D image may not have any barcode visible or only a partial barcodevisible, e.g., where an operator covers all or part of the barcode. Upondetecting the presence of an object being scanned over an allowedscanning distance in the FOV of the 3D imaging apparatus, as determinedat block 604, the block 606 will attempt to identify the object scannedin the 2D image data and in the 3D image data, i.e., by determining afirst and second object identification, respectively. The second objectidentification may be the physical shape, distance, or productclassification (i.e., from a trained object recognition model). If afirst object identification cannot be determined, however, (e.g., whenno barcode can be properly decoded from the 2D image), the block 608determines an improper scanning and generates an alarm signal.

FIG. 13 is a flowchart is shown of an example process 700 for symbologyreading using captured 3D images as may be performed by the scanningstation 302 of FIG. 7 and any of the imaging systems 100, 100′, 100″,140, 140″ and 200 of 1, 2, 3, 4A, 4B, 5, and 6, respectively. Forexample, the process 700 may be implemented by a symbology reader havinga 3D imaging apparatus, with or without having an additional 2D imagingapparatus. At a block 702, a symbology reader of an imaging systemcaptures one or more 3D images through a 3D imaging apparatus associatedwith the symbology reader. In particular, 3D images are captured over anenvironment within a FOV of the 3D imaging apparatus. At a block 704,the imaging system performs facial recognition on the 3D image dataidentifying the presence of facial data within the environment. Thefacial recognition may be performed by examining point cloud data, forexample, and identifying geometric features and comparing them to atrained object identification model, to a 3D anthropometric data modelstored at a scanning station or at a remote server, or to other modelsfor identifying facial features. By using models havingthree-dimensional data, more accurate facial recognition can beperformed.

In response to the identification of the presence of the facial data, ata block 706, the imaging system adjusts one or more operating parametersof the imaging apparatus based on the presence of that facial data. Insome examples, the block 706 adjusts one or more operating parameters ofa 2D imaging apparatus within the imaging system. In an example, theblock 706 adjusts operating parameters of a 2D imaging apparatus of ahandheld barcode reader reducing intensity of at least one of anillumination assembly and an aiming assembly that may also be includedin a symbology reader. In another example of a handheld barcode reader,the block 706 may adjust the operating parameter of the 2D imagingapparatus by preventing activation of at least some portion of the 2Dimaging apparatus until a subsequent performing of facial recognition ona subsequent 3D image data associated with a subsequent 3D image failsto identify another presence of facial data in the subsequent 3D imagedata.

In some examples, the process 700 may further include a block 708capturing a 2D image of an object using a 2D imaging apparatus that hasan operating parameter that has been adjusted at the block 706. Theimaging system may then decode a barcode in the captured 2D image toidentify an object.

In some examples, the block 704 identifies the presence of the facialdata in the 3D image data by determining a position of the facial datain a FOV of the 3D imaging apparatus, and the block 706 adjusts theoperating parameters of the 2D imaging apparatus by adjusting theoperating parameters based on the position of the facial data. In someexamples, the operating parameter adjusted at block 706 is a FOV of the2D imaging apparatus. For example, the FOV of the second imagingapparatus can be adjusted to exclude locations where the facial datawould reside in environment of the FOV. In some examples, the operatingparameter adjusted at block 706 is a focal distance of the 2D imagingapparatus, for example to adjust where the 2D imaging apparatus focusesfor capturing 2D images of an object and/or for identifying a barcode inan object. In some examples, the operating parameter is an exposuretime, an illumination pulse duration, a focus position, or an imagingzoom level of the 2D imaging apparatus.

FIG. 14A illustrates an example 3D image of a first environment of a FOVof the 3D imaging apparatus. FIG. 14B illustrates an example 3D of asecond environment. In the first environment of FIG. 14A, a face 750 isrecognized at a first distance adjacent a platter 752 of a symbologyreader, whereas in the second environment of FIG. 14B, a face 754 isrecognized at a second distance far from the platter 752. Based on theidentification of the facial data at the block 704, for the 3D imagecapturing the environment in FIG. 14A, the block 706, identifying aface, such as the face of a child standing at bi-optic reader height,may adjust an operating parameter of a 2D imaging apparatus to protectthe person from being illuminated by light from an illuminationassembly. For example, the block 706, responding to the environment ofFIG. 14B, may disable the symbology reader from performing a barcodescanning attempt. The block 706 may adjust an exposure time, anillumination pulse duration, or an imaging zoom level of a 2D imagingapparatus to protect the person before passing control to block 708 forcapturing a 2D image.

A similar process may be performed using 2D image data and facialrecognition. FIG. 15 is a flowchart showing an example process 780. Atblock 782, 2D images are captured by a 2D imaging assembly, and at ablock 784 facial recognition is performed on the 2D images. For example,an imaging processor may be configured to use a 2D anthropometric datamodel stored at a scanning station to determine if edge data or othercontrast identified in the 2D image corresponds to facial data. At ablock 786, 3D image data is captured, and a block 788 compares the 2Dimage data and the 3D image data to identify 3D image featuresassociated with the facial data in the 2D image data. If 3D features areidentified, then a block 790 performs one or more different functions,in particular, determining a distance of the facial data from thebarcode reader and selectively disabling/enabling scanning of thebarcode reader based on the distance, determining anthropometric datafor the facial data and determining if facial data is from a person ornot, and adjust at least one operating parameter of the 2D imagingapparatus within the barcode reader. As with the process 700, exampleoperating parameters include an exposure time, an illumination pulseduration, a focal distance, or an imaging zoom level.

FIG. 16 is a flowchart is shown of an example process 800 that mayperformed by a scanning station, such as at a point-of-sale, thatscanning station having 3D imaging apparatus associated therewith, asmay be performed by the scanning station 302 of FIG. 7 and any of theimaging systems 100, 100′, 100″, 140, 140″ and 200 of 1, 2, 3, 4A, 4B,5, and 6, respectively. For example, the process 800 may be implementedby a symbology reader associated with a 3D imaging apparatus, with orwithout having an additional 2D imaging apparatus. At a block 802,similar to block 702, a symbology reader of an imaging system capturesone or more 3D images through a 3D imaging apparatus of the symbologyreader. In particular, 3D images are captured over an environment withina FOV of the 3D imaging apparatus. At a block 804, similar to block 704,the imaging system performs facial recognition on the 3D image dataidentifying the presence of facial data within the environment.

At a block 806, the imaging system performs facial identification on thefacial data from block 704. In particular, the block 806 attempts toauthenticate facial identification. In response to the facialidentification being authenticated, then at a block 808 the 2D imagingapparatus may be used to capture 2D images of object in an environmentof a FOV of the 2D imaging apparatus. In an example, authenticating thefacial identification is achieved by comparing the facial identificationto an authorized user database. In an example, authenticating the facialidentification is achieved by determining that the facial data is at anacceptable distance from the 3D imaging apparatus. For example, facialdata 750 in FIG. 14A may result in an non-authentication, preventing thecapture of 2D images using a 2D imaging apparatus, whereas the facialdata 754 may resulting in authentication. In some examples, beforeperforming facial recognition on the 3D image data, the block 806 mayidentify environmental features in the 3D image data, the environmentalfeatures being features in the 3D image outside of the object, and theprocess 800 may remove environmental features from the 3D image data.

The block 808 may be configured to perform a number of operations inresponse to authenticating facial data. After capturing 2D images, theblock 808 may capture 2D images using a 2D imaging apparatus of thescanning station. That is, in some examples, the block 808 is configuredto allow 2D images and object scanning in response to facial dataauthentication. In some examples, the block 808 may be configured toprevent subsequent 2D image capture. For example, the block 808 mayauthenticate the facial data and satisfy a release condition thatprevents a decode of a barcode captured in image of 2D image. Or theblock 808 may allow a 2D image and barcode decoding to occur, but theblock may prevent adding a subsequently scanned item to a transactionlog of scanned items at the point-of-sale. For example, if facial dataindicates than un-authorized user, e.g., a minor, is attempting to scanan image, the block 808 may prevent a 2D image from being captured orallow a 2D image to be captured but prevent decoding of any barcode inthe image. In some examples, the authentication may be agnostic andprevent any decoding of a barcode in a 2D image. In some examples, theauthentication may prevent such decoding only for certain types ofobjects, such as alcohol or other age inappropriate items. In suchexamples, the block 808 may allow for a 2D image to be capture, abarcode to be identified and decoded, and then depending on the objectidentified by that decoding, determine if the release condition issatisfied such that the decoded barcode is not added to a transactionlog at the point-of-sale scanning station, thereby preventing the minorfrom affecting the scanning and purchase of the object.

While the example processes of FIGS. 15 and 16 are described in thecontext of a symbology reader and scanning station, the processesdescribed may be implemented in any image based authentication system orfraud detection system, such as an automated kiosk for purchasingdispensed items, an automated cash station or teller, or other system.

In various examples, the techniques of the present invention can beimplemented in other scanning application, including, for example, inmachine vision applications. In FIG. 17, a flowchart is shown of anexample process 900 for object scanner, e.g., as may be performed by amachine vision system, using captured 2D images and captured 3D imagesas may be performed by a machine vision system such as the scanningstation 302 implemented as a machine vision system. FIG. 18 illustratesan example logic circuit implemented at an example machine vision system1000. In the illustrated example, the machine vision system 1000includes a 2D color image sensor array 1002 generally configured tosense 2D image data within a FOV of a 2D imaging apparatus. Moreparticularly, the color image sensor array 1002 may be associated with acolor filter array 1004 that includes color filters respectivelyassociated with the individual image sensors of the color image sensorarray 1002. For example, a first image sensor of the color image sensorarray 1002 may be associated with a green filter and a second imagesensor of the color image sensor array 1002 may be associated with a redfilter. The pattern of color filters that form the color filter array1004 may be a Bayer pattern. As illustrated, the machine vision system1000 also includes an illumination assembly 1014 configured to producean illumination light directed toward the imaging FOV of the 2D colorimage sensor array 1004. For example, the illumination assembly 1014 mayinclude one or more light emitting diodes (LEDs) or other types of lightsources. In some embodiments, the illumination assembly 1014 isconfigured to emit white light (e.g., light that includes wavelengthsacross the entire visible spectrum). The illumination assembly 1014 mayinclude multiple illumination sources, such as a direct illuminationsource and a diffuse illumination source. In some examples, theillumination assembly 1014 includes illumination sources emitting atdifferent wavelengths, such as a red light illumination source, a greenlight illumination source, a blue light illumination source, and/or awhite light illumination source. As such, in various examples, the 2Dcolor image sensor array 1002 is capable of sensing the full range oflight reflections. The machine vision system 1000 includes a 3D imagingapparatus 1005 for capturing 3D images of an environment in a FOV of the3D imaging apparatus.

Further, the machine vision system 1000 includes one or more imageprocessors 1006 capable of executing instructions to, for example,implement operations of the example methods described herein, as may berepresented by the flowcharts of the drawings that accompany thisdescription. The image processors 1006 may be one or moremicroprocessors, controllers, and/or any suitable type of processor. Theexample machine vision system 1000 includes a memory (e.g., volatilememory, non-volatile memory) 1008 accessible by the image processors1006 (e.g., via a memory controller). The example machine vision system1000 also includes a decoder 1010 and a machine vision module 1012configured to analyze 2D and 3D image data including image data thathave been processed by the image processors 1006. The example decoder1010 is configured to determine whether the 2D image data from the 2Dcolor image sensor array 1002 is representative of a barcode, and if so,decode the barcode to determine the encoded information. The examplemachine vision module 1012 is configured to perform object recognitiontechniques on the 2D image data and/or 3D image data to identify targetfeatures thereof. For example, the machine vision module 1012 may beconfigured to detect target features such as cracks on an object and/oran incomplete soldering connection for a pin of a microchip asdetermined from either or both of 2D image data and 3D image data. Insome examples, the machine vision module 1012 may be configured todetect features using 2D image data enhanced with 3D image data or 3Dimage data enhanced using 2D image data.

The example machine vision system 1000 also includes a network interface1016 to enable communication with other machines via, for example, oneor more networks, and input/output (I/O) interfaces 1018 to enablereceipt of user input and communication of output data to the user. Forexample, the output data may be the encoded information determined bythe decoder 1010 and/or an indication of the features detected by themachine vision module 1012.

Returning to FIG. 17, at a block 902, a 2D imaging apparatus of amachine vision system (such as the 2D color image sensory array 1002)captures a 2D image of an object over a FOV. The machine vision systemidentifies a barcode of the object in the 2D image and determines one ormore 3D object features of the object from the barcode. For example, theblock 902 may decode the barcode, identify an associated object, anddetermine 3D features of the object, such as geometric features, such asthe shape, surfaces, and/or dimensions of the object. At a block 904, a3D imaging apparatus of the machine vision system (such as the 3Dimaging apparatus 1005) captures 3D image(s) of an environment over aFOV and stores the 3D image data corresponding to the 3D image(s). At ablock 906, the 3D image data is examined by the machine vision system(e.g., by the machine vision module 1012) to identify the presence ofone or more 3D object features, such as those discussed hereinabove. Inresponse to determining the presence of one or more 3D object features,a block 908 provides a digital fault detection signal to a user of themachine vision system. In the illustrated example, if the block 908determines that the 3D object features determined from the block 902 arenot present in the 3D image data from block 906, a digital faultdetection signal is provided to a user. In an example, determining 3Dobject features from the barcode in the 2D image is performed bydecoding the barcode to generate barcode payload data and determiningobject identification from the barcode payload data and determining,from the objecting identification, the one or more 3D object features ofthe object. In another example, determining the 3D object features ofthe object from the barcode in the 2D image is performed by determiningan orientation of the object from a location of the barcode in the 2Dimage and determining, from the orientation of the object, the one ormore 3D object features as a subset of available 3D object features. Insome examples, the 3D object features include at least one of adimensional feature and a shape feature.

If no fault is detected at block 908, at block 910, if it is determinedthat the 3D object features determined from the block 902 are present inthe 3D image data from block 906, the block 910 may determine if achange is to be made to at least one parameter associated with themachine vision system. In this way, the block 910 may use color datafrom a 2D color image sensor array to perform color checking from whichthe block 910 determines if an illumination source should be changed tobetter illuminate an object under inspection by the machine visionsystem. The block 910 may assess 2D image data and 3D image data todetermine properties such as the orientation, lighting, and position ofan object and then adjust the illumination source, the direction of theillumination, the wavelength of the illumination source (e.g., usingillumination sources of different wavelengths). For example, the block910 and determine, optimal illumination conditions, such as illuminationcolor, intensity, and/or direction values for a ‘golden unit’ machinevision scanning and use change parameters to those optimal illuminationconditions. In this way, color data, such as the presence of the colorred determined from 2D image data can be used to adjusting illuminationconditions for performing machine vision scans with the 2D imagingapparatus or the 3D imaging apparatus. In other examples, the color datamay be determined from 3D image data captured by the 3D imagingapparatus. The process 900 allows a machine vision system to captureddata from 2D imaging apparatus and a 3D imaging apparatus, compare theresulting 2D image data and 3D image data and determine changes inoperating parameters of the machine vision system to quickly, on thefly, adjust the machine vision system to capture images under improvedconditions thereby increasing accuracy and scanning throughput of thesesystems.

The above description refers to a block diagram of the accompanyingdrawings. Alternative implementations of the example represented by theblock diagram includes one or more additional or alternative elements,processes and/or devices. Additionally or alternatively, one or more ofthe example blocks of the diagram may be combined, divided, re-arrangedor omitted. Components represented by the blocks of the diagram areimplemented by hardware, software, firmware, and/or any combination ofhardware, software and/or firmware. In some examples, at least one ofthe components represented by the blocks is implemented by a logiccircuit. As used herein, the term “logic circuit” is expressly definedas a physical device including at least one hardware componentconfigured (e.g., via operation in accordance with a predeterminedconfiguration and/or via execution of stored machine-readableinstructions) to control one or more machines and/or perform operationsof one or more machines. Examples of a logic circuit include one or moreprocessors, one or more coprocessors, one or more microprocessors, oneor more controllers, one or more digital signal processors (DSPs), oneor more application specific integrated circuits (ASICs), one or morefield programmable gate arrays (FPGAs), one or more microcontrollerunits (MCUs), one or more hardware accelerators, one or morespecial-purpose computer chips, and one or more system-on-a-chip (SoC)devices. Some example logic circuits, such as ASICs or FPGAs, arespecifically configured hardware for performing operations (e.g., one ormore of the operations described herein and represented by theflowcharts of this disclosure, if such are present). Some example logiccircuits are hardware that executes machine-readable instructions toperform operations (e.g., one or more of the operations described hereinand represented by the flowcharts of this disclosure, if such arepresent). Some example logic circuits include a combination ofspecifically configured hardware and hardware that executesmachine-readable instructions. The above description refers to variousoperations described herein and flowcharts that may be appended heretoto illustrate the flow of those operations. Any such flowcharts arerepresentative of example methods disclosed herein. In some examples,the methods represented by the flowcharts implement the apparatusrepresented by the block diagrams. Alternative implementations ofexample methods disclosed herein may include additional or alternativeoperations. Further, operations of alternative implementations of themethods disclosed herein may combined, divided, re-arranged or omitted.In some examples, the operations described herein are implemented bymachine-readable instructions (e.g., software and/or firmware) stored ona medium (e.g., a tangible machine-readable medium) for execution by oneor more logic circuits (e.g., processor(s)). In some examples, theoperations described herein are implemented by one or moreconfigurations of one or more specifically designed logic circuits(e.g., ASIC(s)). In some examples the operations described herein areimplemented by a combination of specifically designed logic circuit(s)and machine-readable instructions stored on a medium (e.g., a tangiblemachine-readable medium) for execution by logic circuit(s).

As used herein, each of the terms “tangible machine-readable medium,”“non-transitory machine-readable medium” and “machine-readable storagedevice” is expressly defined as a storage medium (e.g., a platter of ahard disk drive, a digital versatile disc, a compact disc, flash memory,read-only memory, random-access memory, etc.) on which machine-readableinstructions (e.g., program code in the form of, for example, softwareand/or firmware) are stored for any suitable duration of time (e.g.,permanently, for an extended period of time (e.g., while a programassociated with the machine-readable instructions is executing), and/ora short period of time (e.g., while the machine-readable instructionsare cached and/or during a buffering process)). Further, as used herein,each of the terms “tangible machine-readable medium,” “non-transitorymachine-readable medium” and “machine-readable storage device” isexpressly defined to exclude propagating signals. That is, as used inany claim of this patent, none of the terms “tangible machine-readablemedium,” “non-transitory machine-readable medium,” and “machine-readablestorage device” can be read to be implemented by a propagating signal.

In the foregoing specification, specific embodiments have beendescribed. However, one of ordinary skill in the art appreciates thatvarious modifications and changes can be made without departing from thescope of the invention as set forth in the claims below. Accordingly,the specification and figures are to be regarded in an illustrativerather than a restrictive sense, and all such modifications are intendedto be included within the scope of present teachings. Additionally, thedescribed embodiments/examples/implementations should not be interpretedas mutually exclusive, and should instead be understood as potentiallycombinable if such combinations are permissive in any way. In otherwords, any feature disclosed in any of the aforementionedembodiments/examples/implementations may be included in any of the otheraforementioned embodiments/examples/implementations.

The benefits, advantages, solutions to problems, and any element(s) thatmay cause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeatures or elements of any or all the claims. The claimed invention isdefined solely by the appended claims including any amendments madeduring the pendency of this application and all equivalents of thoseclaims as issued.

Moreover in this document, relational terms such as first and second,top and bottom, and the like may be used solely to distinguish oneentity or action from another entity or action without necessarilyrequiring or implying any actual such relationship or order between suchentities or actions. The terms “comprises,” “comprising,” “has”,“having,” “includes”, “including,” “contains”, “containing” or any othervariation thereof, are intended to cover a non-exclusive inclusion, suchthat a process, method, article, or apparatus that comprises, has,includes, contains a list of elements does not include only thoseelements but may include other elements not expressly listed or inherentto such process, method, article, or apparatus. An element proceeded by“comprises . . . a”, “has . . . a”, “includes . . . a”, “contains . . .a” does not, without more constraints, preclude the existence ofadditional identical elements in the process, method, article, orapparatus that comprises, has, includes, contains the element. The terms“a” and “an” are defined as one or more unless explicitly statedotherwise herein. The terms “substantially”, “essentially”,“approximately”, “about” or any other version thereof, are defined asbeing close to as understood by one of ordinary skill in the art, and inone non-limiting embodiment the term is defined to be within 10%, inanother embodiment within 5%, in another embodiment within 1% and inanother embodiment within 0.5%. The term “coupled” as used herein isdefined as connected, although not necessarily directly and notnecessarily mechanically. A device or structure that is “configured” ina certain way is configured in at least that way, but may also beconfigured in ways that are not listed.

The Abstract of the Disclosure is provided to allow the reader toquickly ascertain the nature of the technical disclosure. It issubmitted with the understanding that it will not be used to interpretor limit the scope or meaning of the claims. In addition, in theforegoing Detailed Description, it can be seen that various features aregrouped together in various embodiments for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter may lie in less thanall features of a single disclosed embodiment. Thus, the followingclaims are hereby incorporated into the Detailed Description, with eachclaim standing on its own as a separately claimed subject matter.

1. A method of barcode scanning using a barcode reader, the methodcomprising: capturing, using a two-dimensional (2D) imaging apparatuswithin the barcode reader and having a first field of view (FOV), a 2Dimage of a first environment appearing within the first FOV and storing2D image data corresponding to the 2D image; capturing, using athree-dimensional (3D) imaging apparatus associated with the barcodereader and having a second FOV that overlaps at least partially with thefirst FOV, a 3D image of a second environment appearing within thesecond FOV and storing 3D image data corresponding to the 3D image;identifying one or more 3D image features within the second environmentfrom the 3D image data; enhancing the 2D image data, resulting inenhanced 2D image data, by correlating the one or more 3D image featuresto at least one or more 2D image features in the 2D image data; andprocessing the enhanced 2D image data to at least one of (a) decode abarcode captured within the enhanced 2D image data, (b) train an objectrecognition model with the enhanced 2D image data, (c) recognize anobject within the enhanced 2D image data, (d) identify an actionperformed by an operator of the barcode reader, and (e) change at leastone parameter associated with the barcode reader operation.
 2. Themethod of claim 1, wherein the 3D image data comprises 3D point clouddata and the one or more 3D image features includes one or moregeometric features of an object presented within the second FOV.
 3. Themethod of claim 1, wherein the 3D image data comprises 3D point clouddata and the one or more 3D image features includes a color or a colorgradation corresponding an object presented within the second FOV. 4.The method of claim 1, wherein the identifying the one or more 3D imagefeatures includes identifying the one or more 3D image features that arepositioned within a predetermined distance range away from the 3Dimaging apparatus.
 5. The method of claim 4, wherein the enhancing the2D image data further includes filtering the at least one or more 2Dimage features such that the processing the enhanced 2D image dataexcludes processing of image data associated with the at least one ormore 2D image features.
 6. The method of claim 4, wherein the enhancingthe 2D image data further includes filtering the at least one or more 2Dimage features such that the processing the enhanced 2D image data islimited to processing of image data associated with the at least one ormore 2D image features.
 7. The method of claim 4, wherein the barcodereader is a stationary barcode reader configured to be positioned withina workstation and operated by the operator, and wherein thepredetermined distance range away from the 3D imaging apparatus extendsfrom the 3D imaging apparatus to an edge of the workstation proximatethe operator.
 8. The method of claim 4, wherein the barcode reader is abi-optic barcode reader having a product-scanning region, and whereinthe predetermined distance range away from the 3D imaging apparatusextends from the 3D imaging apparatus to a distal boundary of theproduct-scanning region.
 9. The method of claim 1, wherein the one ormore 3D image features includes at least one of (i) at least a portionof an operator's hand, and (ii) an object grasped by the operator'shand.
 10. The method of claim 1, wherein the processing the enhanced 2Dimage data includes the identifying the action performed by theoperator, and responsive to the action performed by the operator beingidentified as one of presenting an object within a product-scanningregion and presenting the object proximate to the product-scanningregion, and further responsive to no barcode being detected within atleast one of the 2D image data and the enhanced 2D image data,generating an alert suitable for signaling a potential theft event. 11.The method of claim 1, wherein the processing the enhanced 2D image dataincludes the identifying the action performed by the operator, andresponsive to the action performed by the operator being identified asone of presenting an object within a product-scanning region andpresenting the object proximate to the product-scanning region, anddetecting a partially covered barcode or fully covered barcode on theobject within at least one of the 2D image data and the enhanced 2Dimage data, generating an alert suitable for signaling a potential theftevent.
 12. The method of claim 1, wherein the at least one parameterassociated with the barcode reader operation is an exposure time of thebarcode reader, an illumination pulse duration of the barcode reader, afocus position of the barcode reader, an imaging zoom level of thebarcode reader, and an illumination source of the barcode reader. 13.The method of claim 12, wherein the illumination source is a diffuseillumination source or a direct illumination source.
 14. The method ofclaim 1, wherein, in response to identifying the one or more 3D imagefeatures within the second environment from the 3D image data, adjustingan illumination brightness of the barcode reader prior to capturing the2D image of the first environment.
 15. A method processing data using abarcode reader, the method comprising: capturing, using atwo-dimensional (2D) imaging apparatus within the barcode reader andhaving a first field of view (FOV), a 2D image of a first environmentappearing within the first FOV and storing 2D image data correspondingto the 2D image; capturing, using a three-dimensional (3D) imagingapparatus associated with the barcode reader and having a second FOVthat overlaps at least partially with the first FOV, a 3D image of asecond environment appearing within the second FOV and storing 3D imagedata corresponding to the 3D image; identifying one or more 2D imagefeatures within the first environment from the 2D image data; enhancingthe 3D image data, resulting in enhanced 3D image data, by correlatingthe one or more 2D image features to at least one or more 3D imagefeatures in the 3D image data; and processing the enhanced 3D image datato at least one of (a) train an object recognition model with theenhanced 3D image data, (b) recognize an object within the enhanced 3Dimage data, (c) identify an action performed by a user of the barcodereader, and (d) change at least one parameter associated with the 3Dimaging apparatus.
 16. The method of claim 15, wherein the 2D image datacomprises one of monochrome image data, grayscale image data, andpolychrome image data, and wherein the one or more 2D image featuresincludes at least one of a barcode and one or more geometric features ofan object presented within the first FOV.
 17. The method of claim 16,wherein the one or more 2D image features includes the barcode, andwherein the enhancing the 3D image data includes mapping a location ofthe barcode from the 2D image data to the 3D image data.
 18. The methodof claim 16, wherein the 2D image data comprises polychrome image data,wherein the one or more 2D image features includes the one or moregeometric features of the object presented within the first FOV, andwherein the enhancing the 3D image data includes mapping at least aportion of the polychrome image data to the 3D image data based at leastin part on the one or more geometric features of the object presentedwithin the first FOV.
 19. The method of claim 16, wherein the enhancingthe 3D image data further includes filtering the at least one or more 3Dimage features such that the processing the enhanced 3D image dataexcludes processing of image data associated with the at least one ormore 3D image features.
 20. The method of claim 16, wherein theenhancing the 3D image data further includes filtering the at least oneor more 3D image features such that the processing the enhanced 3D imagedata is limited to processing of image data associated with the at leastone or more 3D image features.
 21. The method of claim 16, wherein theenhancing the 3D image data further includes filtering the 3D image databased on a predetermined distance range away from the 3D imagingapparatus.
 22. The method of claim 21, wherein the barcode reader is astationary barcode reader configured to be positioned within aworkstation and operated by the operator, and wherein the predetermineddistance range away from the 3D imaging apparatus extends from the 3Dimaging apparatus to an edge of the workstation proximate the operator.23. The method of claim 21, wherein the barcode reader is a bi-opticbarcode reader having a product-scanning region, and wherein thepredetermined distance range away from the 3D imaging apparatus extendsfrom the 3D imaging apparatus to a distal boundary of theproduct-scanning region.
 24. The method of claim 15, wherein the one ormore 2D image features includes at least one of (i) at least a portionof an operator's hand, and (ii) an object grasped by the operator'shand.
 25. The method of claim 15, wherein the processing the enhanced 3Dimage data includes the identifying the action performed by theoperator, and responsive to the action performed by the operator beingidentified as one of presenting an object within a product-scanningregion and presenting the object proximate to the product-scanningregion, and further responsive to no barcode being detected within the2D image data generating an alert suitable for signaling a potentialtheft event.
 26. The method of claim 15, wherein identifying the one ormore 2D image features comprises: identifying environmental features onthe 2D image, wherein the environmental features are features in theimage outside of an object presented within the first FOV; convertingthe environmental features into masking features configured to cover theidentified environmental features in the 2D image; and identifying themasking features as the one or more 2D image features.
 27. The method ofclaim 15, wherein identifying one or more 2D image features comprises:identifying a barcode for the object in the 2D image data; decoding thebarcode to generate barcode payload data and determining objectidentification from the barcode payload data; and determining, from theobjecting identification, the one or more 2D image features.
 28. Themethod of claim 15, wherein processing the enhanced 3D image data totrain the object recognition model with the enhanced 3D image datacomprises: identifying a barcode in the 2D image data, determining abarcode detection event timeframe, and training the object recognitionmodel with the enhanced 3D image data corresponding to the barcodedetection event timeframe; or identifying a barcode in the 2D imagedata, identifying the object within the enhanced 3D image datacorresponding to the barcode in the 2D image data, and upon identifyingother object in the 3D image not corresponding to the barcode, removingthe other objects from the enhanced 3D image data before training theobject recognition model with the enhanced 3D image data.
 29. The methodof claim 15, wherein the at least one parameter associated with the 3Dimaging apparatus comprises a projected amount of illumination of the 3Dimaging apparatus, a projected direction of illumination of the 3Dimaging apparatus, or a source of illumination for the 3D imagingapparatus. 30-107. (canceled)